2025-02-14 05:02:58,144 - training_args.py:2100 - _setup_devices - INFO - PyTorch: setting up devices 2025-02-14 05:02:58,680 - configuration_utils.py:731 - _get_config_dict - INFO - loading configuration file ./checkpoints/longvu_llama3_2/config.json 2025-02-14 05:02:58,683 - configuration_utils.py:800 - from_dict - INFO - Model config CambrianConfig { "_name_or_path": "/tmp/iopath_cache/manifold_cache/tree/users/shenx/finetune/09281004-cambrian_llama3_2_t576_ov", "architectures": [ "CambrianLlamaForCausalLM" ], "attention_bias": false, "attention_dropout": 0.0, "bos_token_id": 128000, "connect_layer": 2, "connector_depth": 3, "connector_only": true, "dino_threshold": 0.83, "drop_threshold": 0.8, "eos_token_id": [ 128001, 128008, 128009 ], "frame_pos": false, "freeze_mm_mlp_adapter": false, "hidden_act": "silu", "hidden_size": 3072, "highres": true, "highres_connect": false, "image_aspect_ratio": "pad", "image_position": 91, "image_token_len": 144, "initializer_range": 0.02, "intermediate_size": 8192, "is_image_newline": true, "is_st_sampler": false, "lowres_token": 8, "max_position_embeddings": 131072, "mlp_bias": false, "mm_patch_merge_type": "flat", "mm_projector_lr": null, "mm_projector_type": "sva", "mm_use_im_patch_token": false, "mm_use_im_start_end": false, "mm_vision_sampler_lr": null, "mm_vision_select_feature": "patch", "mm_vision_select_layer": -2, "mm_vision_tower_aux_list": [ "siglip/CLIP-ViT-SO400M-14-384", "facebook/dinov2-giant-res378" ], "mm_vision_tower_aux_token_len_list": [ 576, 576 ], "mm_vision_tower_lr": null, "model_type": "cambrian_llama", "num_attention_heads": 24, "num_hidden_layers": 28, "num_key_value_heads": 8, "num_of_vision_sampler_layers": 10, "num_query_group": 1, "pretraining_tp": 1, "query_num_list": [ 144 ], "rms_norm_eps": 1e-05, "rope_scaling": { "factor": 32.0, "high_freq_factor": 4.0, "low_freq_factor": 1.0, "original_max_position_embeddings": 8192, "rope_type": "llama3" }, "rope_theta": 500000.0, "spmd_debug": null, "spmd_fsdp_sharding": null, "spmd_mesh": null, "start_of_vision_sampler_layers": 0, "stride_of_vision_sampler_layers": 3, "tie_word_embeddings": false, "tokenizer_model_max_length": 8192, "tokenizer_padding_side": "right", "torch_dtype": "float32", "transformers_version": "4.43.1", "tune_mm_mlp_adapter": false, "unfreeze_mm_vision_tower": false, "use_cache": false, "use_mm_proj": true, "vision_hidden_size": 1024, "vision_tower_aux_token_len_list": [ 576, 576 ], "vocab_size": 128256 } 2025-02-14 05:02:58,684 - modeling_utils.py:3618 - from_pretrained - INFO - loading weights file ./checkpoints/longvu_llama3_2/pytorch_model.bin 2025-02-14 05:02:58,722 - configuration_utils.py:1038 - from_dict - INFO - Generate config GenerationConfig { "bos_token_id": 128000, "eos_token_id": [ 128001, 128008, 128009 ], "use_cache": false } 2025-02-14 05:02:58,929 - configuration_utils.py:733 - _get_config_dict - INFO - loading configuration file config.json from cache at /root/.cache/huggingface/hub/models--facebook--dinov2-giant/snapshots/611a9d42f2335e0f921f1e313ad3c1b7178d206d/config.json 2025-02-14 05:02:58,933 - configuration_utils.py:800 - from_dict - INFO - Model config Dinov2Config { "apply_layernorm": true, "architectures": [ "Dinov2Model" ], "attention_probs_dropout_prob": 0.0, "drop_path_rate": 0.0, "hidden_act": "gelu", "hidden_dropout_prob": 0.0, "hidden_size": 1536, "image_size": 518, "initializer_range": 0.02, "layer_norm_eps": 1e-06, "layerscale_value": 1.0, "mlp_ratio": 4, "model_type": "dinov2", "num_attention_heads": 24, "num_channels": 3, "num_hidden_layers": 40, "out_features": [ "stage40" ], "out_indices": [ 40 ], "patch_size": 14, "qkv_bias": true, "reshape_hidden_states": true, "stage_names": [ "stem", "stage1", "stage2", "stage3", "stage4", "stage5", "stage6", "stage7", "stage8", "stage9", "stage10", "stage11", "stage12", "stage13", "stage14", "stage15", "stage16", "stage17", "stage18", "stage19", "stage20", "stage21", "stage22", "stage23", "stage24", "stage25", "stage26", "stage27", "stage28", "stage29", "stage30", "stage31", "stage32", "stage33", "stage34", "stage35", "stage36", "stage37", "stage38", "stage39", "stage40" ], "torch_dtype": "float32", "transformers_version": "4.43.1", "use_swiglu_ffn": true } 2025-02-14 05:03:00,300 - modeling_utils.py:4450 - _load_pretrained_model - INFO - All model checkpoint weights were used when initializing CambrianLlamaForCausalLM. 2025-02-14 05:03:00,301 - modeling_utils.py:4458 - _load_pretrained_model - INFO - All the weights of CambrianLlamaForCausalLM were initialized from the model checkpoint at ./checkpoints/longvu_llama3_2. If your task is similar to the task the model of the checkpoint was trained on, you can already use CambrianLlamaForCausalLM for predictions without further training. 2025-02-14 05:03:00,304 - configuration_utils.py:991 - from_pretrained - INFO - loading configuration file ./checkpoints/longvu_llama3_2/generation_config.json 2025-02-14 05:03:00,305 - configuration_utils.py:1038 - from_dict - INFO - Generate config GenerationConfig { "bos_token_id": 128000, "do_sample": true, "eos_token_id": [ 128001, 128008, 128009 ], "temperature": 0.6, "top_p": 0.9 } 2025-02-14 05:03:00,459 - tokenization_utils_base.py:2287 - from_pretrained - INFO - loading file tokenizer.json 2025-02-14 05:03:00,459 - tokenization_utils_base.py:2287 - from_pretrained - INFO - loading file added_tokens.json 2025-02-14 05:03:00,459 - tokenization_utils_base.py:2287 - from_pretrained - INFO - loading file special_tokens_map.json 2025-02-14 05:03:00,459 - tokenization_utils_base.py:2287 - from_pretrained - INFO - loading file tokenizer_config.json 2025-02-14 05:03:00,684 - tokenization_utils_base.py:2533 - _from_pretrained - INFO - Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained. 2025-02-14 05:03:01,096 - configuration_utils.py:733 - _get_config_dict - INFO - loading configuration file config.json from cache at /root/.cache/huggingface/hub/models--google--siglip-so400m-patch14-384/snapshots/9fdffc58afc957d1a03a25b10dba0329ab15c2a3/config.json 2025-02-14 05:03:01,098 - configuration_utils.py:800 - from_dict - INFO - Model config SiglipVisionConfig { "attention_dropout": 0.0, "hidden_act": "gelu_pytorch_tanh", "hidden_size": 1152, "image_size": 384, "intermediate_size": 4304, "layer_norm_eps": 1e-06, "model_type": "siglip_vision_model", "num_attention_heads": 16, "num_channels": 3, "num_hidden_layers": 27, "patch_size": 14, "transformers_version": "4.43.1" } 2025-02-14 05:03:01,099 - modeling_utils.py:3621 - from_pretrained - INFO - loading weights file model.safetensors from cache at /root/.cache/huggingface/hub/models--google--siglip-so400m-patch14-384/snapshots/9fdffc58afc957d1a03a25b10dba0329ab15c2a3/model.safetensors 2025-02-14 05:03:01,241 - modeling_utils.py:4440 - _load_pretrained_model - INFO - Some weights of the model checkpoint at google/siglip-so400m-patch14-384 were not used when initializing SiglipVisionModel: ['logit_bias', 'logit_scale', 'text_model.embeddings.position_embedding.weight', 'text_model.embeddings.token_embedding.weight', 'text_model.encoder.layers.0.layer_norm1.bias', 'text_model.encoder.layers.0.layer_norm1.weight', 'text_model.encoder.layers.0.layer_norm2.bias', 'text_model.encoder.layers.0.layer_norm2.weight', 'text_model.encoder.layers.0.mlp.fc1.bias', 'text_model.encoder.layers.0.mlp.fc1.weight', 'text_model.encoder.layers.0.mlp.fc2.bias', 'text_model.encoder.layers.0.mlp.fc2.weight', 'text_model.encoder.layers.0.self_attn.k_proj.bias', 'text_model.encoder.layers.0.self_attn.k_proj.weight', 'text_model.encoder.layers.0.self_attn.out_proj.bias', 'text_model.encoder.layers.0.self_attn.out_proj.weight', 'text_model.encoder.layers.0.self_attn.q_proj.bias', 'text_model.encoder.layers.0.self_attn.q_proj.weight', 'text_model.encoder.layers.0.self_attn.v_proj.bias', 'text_model.encoder.layers.0.self_attn.v_proj.weight', 'text_model.encoder.layers.1.layer_norm1.bias', 'text_model.encoder.layers.1.layer_norm1.weight', 'text_model.encoder.layers.1.layer_norm2.bias', 'text_model.encoder.layers.1.layer_norm2.weight', 'text_model.encoder.layers.1.mlp.fc1.bias', 'text_model.encoder.layers.1.mlp.fc1.weight', 'text_model.encoder.layers.1.mlp.fc2.bias', 'text_model.encoder.layers.1.mlp.fc2.weight', 'text_model.encoder.layers.1.self_attn.k_proj.bias', 'text_model.encoder.layers.1.self_attn.k_proj.weight', 'text_model.encoder.layers.1.self_attn.out_proj.bias', 'text_model.encoder.layers.1.self_attn.out_proj.weight', 'text_model.encoder.layers.1.self_attn.q_proj.bias', 'text_model.encoder.layers.1.self_attn.q_proj.weight', 'text_model.encoder.layers.1.self_attn.v_proj.bias', 'text_model.encoder.layers.1.self_attn.v_proj.weight', 'text_model.encoder.layers.10.layer_norm1.bias', 'text_model.encoder.layers.10.layer_norm1.weight', 'text_model.encoder.layers.10.layer_norm2.bias', 'text_model.encoder.layers.10.layer_norm2.weight', 'text_model.encoder.layers.10.mlp.fc1.bias', 'text_model.encoder.layers.10.mlp.fc1.weight', 'text_model.encoder.layers.10.mlp.fc2.bias', 'text_model.encoder.layers.10.mlp.fc2.weight', 'text_model.encoder.layers.10.self_attn.k_proj.bias', 'text_model.encoder.layers.10.self_attn.k_proj.weight', 'text_model.encoder.layers.10.self_attn.out_proj.bias', 'text_model.encoder.layers.10.self_attn.out_proj.weight', 'text_model.encoder.layers.10.self_attn.q_proj.bias', 'text_model.encoder.layers.10.self_attn.q_proj.weight', 'text_model.encoder.layers.10.self_attn.v_proj.bias', 'text_model.encoder.layers.10.self_attn.v_proj.weight', 'text_model.encoder.layers.11.layer_norm1.bias', 'text_model.encoder.layers.11.layer_norm1.weight', 'text_model.encoder.layers.11.layer_norm2.bias', 'text_model.encoder.layers.11.layer_norm2.weight', 'text_model.encoder.layers.11.mlp.fc1.bias', 'text_model.encoder.layers.11.mlp.fc1.weight', 'text_model.encoder.layers.11.mlp.fc2.bias', 'text_model.encoder.layers.11.mlp.fc2.weight', 'text_model.encoder.layers.11.self_attn.k_proj.bias', 'text_model.encoder.layers.11.self_attn.k_proj.weight', 'text_model.encoder.layers.11.self_attn.out_proj.bias', 'text_model.encoder.layers.11.self_attn.out_proj.weight', 'text_model.encoder.layers.11.self_attn.q_proj.bias', 'text_model.encoder.layers.11.self_attn.q_proj.weight', 'text_model.encoder.layers.11.self_attn.v_proj.bias', 'text_model.encoder.layers.11.self_attn.v_proj.weight', 'text_model.encoder.layers.12.layer_norm1.bias', 'text_model.encoder.layers.12.layer_norm1.weight', 'text_model.encoder.layers.12.layer_norm2.bias', 'text_model.encoder.layers.12.layer_norm2.weight', 'text_model.encoder.layers.12.mlp.fc1.bias', 'text_model.encoder.layers.12.mlp.fc1.weight', 'text_model.encoder.layers.12.mlp.fc2.bias', 'text_model.encoder.layers.12.mlp.fc2.weight', 'text_model.encoder.layers.12.self_attn.k_proj.bias', 'text_model.encoder.layers.12.self_attn.k_proj.weight', 'text_model.encoder.layers.12.self_attn.out_proj.bias', 'text_model.encoder.layers.12.self_attn.out_proj.weight', 'text_model.encoder.layers.12.self_attn.q_proj.bias', 'text_model.encoder.layers.12.self_attn.q_proj.weight', 'text_model.encoder.layers.12.self_attn.v_proj.bias', 'text_model.encoder.layers.12.self_attn.v_proj.weight', 'text_model.encoder.layers.13.layer_norm1.bias', 'text_model.encoder.layers.13.layer_norm1.weight', 'text_model.encoder.layers.13.layer_norm2.bias', 'text_model.encoder.layers.13.layer_norm2.weight', 'text_model.encoder.layers.13.mlp.fc1.bias', 'text_model.encoder.layers.13.mlp.fc1.weight', 'text_model.encoder.layers.13.mlp.fc2.bias', 'text_model.encoder.layers.13.mlp.fc2.weight', 'text_model.encoder.layers.13.self_attn.k_proj.bias', 'text_model.encoder.layers.13.self_attn.k_proj.weight', 'text_model.encoder.layers.13.self_attn.out_proj.bias', 'text_model.encoder.layers.13.self_attn.out_proj.weight', 'text_model.encoder.layers.13.self_attn.q_proj.bias', 'text_model.encoder.layers.13.self_attn.q_proj.weight', 'text_model.encoder.layers.13.self_attn.v_proj.bias', 'text_model.encoder.layers.13.self_attn.v_proj.weight', 'text_model.encoder.layers.14.layer_norm1.bias', 'text_model.encoder.layers.14.layer_norm1.weight', 'text_model.encoder.layers.14.layer_norm2.bias', 'text_model.encoder.layers.14.layer_norm2.weight', 'text_model.encoder.layers.14.mlp.fc1.bias', 'text_model.encoder.layers.14.mlp.fc1.weight', 'text_model.encoder.layers.14.mlp.fc2.bias', 'text_model.encoder.layers.14.mlp.fc2.weight', 'text_model.encoder.layers.14.self_attn.k_proj.bias', 'text_model.encoder.layers.14.self_attn.k_proj.weight', 'text_model.encoder.layers.14.self_attn.out_proj.bias', 'text_model.encoder.layers.14.self_attn.out_proj.weight', 'text_model.encoder.layers.14.self_attn.q_proj.bias', 'text_model.encoder.layers.14.self_attn.q_proj.weight', 'text_model.encoder.layers.14.self_attn.v_proj.bias', 'text_model.encoder.layers.14.self_attn.v_proj.weight', 'text_model.encoder.layers.15.layer_norm1.bias', 'text_model.encoder.layers.15.layer_norm1.weight', 'text_model.encoder.layers.15.layer_norm2.bias', 'text_model.encoder.layers.15.layer_norm2.weight', 'text_model.encoder.layers.15.mlp.fc1.bias', 'text_model.encoder.layers.15.mlp.fc1.weight', 'text_model.encoder.layers.15.mlp.fc2.bias', 'text_model.encoder.layers.15.mlp.fc2.weight', 'text_model.encoder.layers.15.self_attn.k_proj.bias', 'text_model.encoder.layers.15.self_attn.k_proj.weight', 'text_model.encoder.layers.15.self_attn.out_proj.bias', 'text_model.encoder.layers.15.self_attn.out_proj.weight', 'text_model.encoder.layers.15.self_attn.q_proj.bias', 'text_model.encoder.layers.15.self_attn.q_proj.weight', 'text_model.encoder.layers.15.self_attn.v_proj.bias', 'text_model.encoder.layers.15.self_attn.v_proj.weight', 'text_model.encoder.layers.16.layer_norm1.bias', 'text_model.encoder.layers.16.layer_norm1.weight', 'text_model.encoder.layers.16.layer_norm2.bias', 'text_model.encoder.layers.16.layer_norm2.weight', 'text_model.encoder.layers.16.mlp.fc1.bias', 'text_model.encoder.layers.16.mlp.fc1.weight', 'text_model.encoder.layers.16.mlp.fc2.bias', 'text_model.encoder.layers.16.mlp.fc2.weight', 'text_model.encoder.layers.16.self_attn.k_proj.bias', 'text_model.encoder.layers.16.self_attn.k_proj.weight', 'text_model.encoder.layers.16.self_attn.out_proj.bias', 'text_model.encoder.layers.16.self_attn.out_proj.weight', 'text_model.encoder.layers.16.self_attn.q_proj.bias', 'text_model.encoder.layers.16.self_attn.q_proj.weight', 'text_model.encoder.layers.16.self_attn.v_proj.bias', 'text_model.encoder.layers.16.self_attn.v_proj.weight', 'text_model.encoder.layers.17.layer_norm1.bias', 'text_model.encoder.layers.17.layer_norm1.weight', 'text_model.encoder.layers.17.layer_norm2.bias', 'text_model.encoder.layers.17.layer_norm2.weight', 'text_model.encoder.layers.17.mlp.fc1.bias', 'text_model.encoder.layers.17.mlp.fc1.weight', 'text_model.encoder.layers.17.mlp.fc2.bias', 'text_model.encoder.layers.17.mlp.fc2.weight', 'text_model.encoder.layers.17.self_attn.k_proj.bias', 'text_model.encoder.layers.17.self_attn.k_proj.weight', 'text_model.encoder.layers.17.self_attn.out_proj.bias', 'text_model.encoder.layers.17.self_attn.out_proj.weight', 'text_model.encoder.layers.17.self_attn.q_proj.bias', 'text_model.encoder.layers.17.self_attn.q_proj.weight', 'text_model.encoder.layers.17.self_attn.v_proj.bias', 'text_model.encoder.layers.17.self_attn.v_proj.weight', 'text_model.encoder.layers.18.layer_norm1.bias', 'text_model.encoder.layers.18.layer_norm1.weight', 'text_model.encoder.layers.18.layer_norm2.bias', 'text_model.encoder.layers.18.layer_norm2.weight', 'text_model.encoder.layers.18.mlp.fc1.bias', 'text_model.encoder.layers.18.mlp.fc1.weight', 'text_model.encoder.layers.18.mlp.fc2.bias', 'text_model.encoder.layers.18.mlp.fc2.weight', 'text_model.encoder.layers.18.self_attn.k_proj.bias', 'text_model.encoder.layers.18.self_attn.k_proj.weight', 'text_model.encoder.layers.18.self_attn.out_proj.bias', 'text_model.encoder.layers.18.self_attn.out_proj.weight', 'text_model.encoder.layers.18.self_attn.q_proj.bias', 'text_model.encoder.layers.18.self_attn.q_proj.weight', 'text_model.encoder.layers.18.self_attn.v_proj.bias', 'text_model.encoder.layers.18.self_attn.v_proj.weight', 'text_model.encoder.layers.19.layer_norm1.bias', 'text_model.encoder.layers.19.layer_norm1.weight', 'text_model.encoder.layers.19.layer_norm2.bias', 'text_model.encoder.layers.19.layer_norm2.weight', 'text_model.encoder.layers.19.mlp.fc1.bias', 'text_model.encoder.layers.19.mlp.fc1.weight', 'text_model.encoder.layers.19.mlp.fc2.bias', 'text_model.encoder.layers.19.mlp.fc2.weight', 'text_model.encoder.layers.19.self_attn.k_proj.bias', 'text_model.encoder.layers.19.self_attn.k_proj.weight', 'text_model.encoder.layers.19.self_attn.out_proj.bias', 'text_model.encoder.layers.19.self_attn.out_proj.weight', 'text_model.encoder.layers.19.self_attn.q_proj.bias', 'text_model.encoder.layers.19.self_attn.q_proj.weight', 'text_model.encoder.layers.19.self_attn.v_proj.bias', 'text_model.encoder.layers.19.self_attn.v_proj.weight', 'text_model.encoder.layers.2.layer_norm1.bias', 'text_model.encoder.layers.2.layer_norm1.weight', 'text_model.encoder.layers.2.layer_norm2.bias', 'text_model.encoder.layers.2.layer_norm2.weight', 'text_model.encoder.layers.2.mlp.fc1.bias', 'text_model.encoder.layers.2.mlp.fc1.weight', 'text_model.encoder.layers.2.mlp.fc2.bias', 'text_model.encoder.layers.2.mlp.fc2.weight', 'text_model.encoder.layers.2.self_attn.k_proj.bias', 'text_model.encoder.layers.2.self_attn.k_proj.weight', 'text_model.encoder.layers.2.self_attn.out_proj.bias', 'text_model.encoder.layers.2.self_attn.out_proj.weight', 'text_model.encoder.layers.2.self_attn.q_proj.bias', 'text_model.encoder.layers.2.self_attn.q_proj.weight', 'text_model.encoder.layers.2.self_attn.v_proj.bias', 'text_model.encoder.layers.2.self_attn.v_proj.weight', 'text_model.encoder.layers.20.layer_norm1.bias', 'text_model.encoder.layers.20.layer_norm1.weight', 'text_model.encoder.layers.20.layer_norm2.bias', 'text_model.encoder.layers.20.layer_norm2.weight', 'text_model.encoder.layers.20.mlp.fc1.bias', 'text_model.encoder.layers.20.mlp.fc1.weight', 'text_model.encoder.layers.20.mlp.fc2.bias', 'text_model.encoder.layers.20.mlp.fc2.weight', 'text_model.encoder.layers.20.self_attn.k_proj.bias', 'text_model.encoder.layers.20.self_attn.k_proj.weight', 'text_model.encoder.layers.20.self_attn.out_proj.bias', 'text_model.encoder.layers.20.self_attn.out_proj.weight', 'text_model.encoder.layers.20.self_attn.q_proj.bias', 'text_model.encoder.layers.20.self_attn.q_proj.weight', 'text_model.encoder.layers.20.self_attn.v_proj.bias', 'text_model.encoder.layers.20.self_attn.v_proj.weight', 'text_model.encoder.layers.21.layer_norm1.bias', 'text_model.encoder.layers.21.layer_norm1.weight', 'text_model.encoder.layers.21.layer_norm2.bias', 'text_model.encoder.layers.21.layer_norm2.weight', 'text_model.encoder.layers.21.mlp.fc1.bias', 'text_model.encoder.layers.21.mlp.fc1.weight', 'text_model.encoder.layers.21.mlp.fc2.bias', 'text_model.encoder.layers.21.mlp.fc2.weight', 'text_model.encoder.layers.21.self_attn.k_proj.bias', 'text_model.encoder.layers.21.self_attn.k_proj.weight', 'text_model.encoder.layers.21.self_attn.out_proj.bias', 'text_model.encoder.layers.21.self_attn.out_proj.weight', 'text_model.encoder.layers.21.self_attn.q_proj.bias', 'text_model.encoder.layers.21.self_attn.q_proj.weight', 'text_model.encoder.layers.21.self_attn.v_proj.bias', 'text_model.encoder.layers.21.self_attn.v_proj.weight', 'text_model.encoder.layers.22.layer_norm1.bias', 'text_model.encoder.layers.22.layer_norm1.weight', 'text_model.encoder.layers.22.layer_norm2.bias', 'text_model.encoder.layers.22.layer_norm2.weight', 'text_model.encoder.layers.22.mlp.fc1.bias', 'text_model.encoder.layers.22.mlp.fc1.weight', 'text_model.encoder.layers.22.mlp.fc2.bias', 'text_model.encoder.layers.22.mlp.fc2.weight', 'text_model.encoder.layers.22.self_attn.k_proj.bias', 'text_model.encoder.layers.22.self_attn.k_proj.weight', 'text_model.encoder.layers.22.self_attn.out_proj.bias', 'text_model.encoder.layers.22.self_attn.out_proj.weight', 'text_model.encoder.layers.22.self_attn.q_proj.bias', 'text_model.encoder.layers.22.self_attn.q_proj.weight', 'text_model.encoder.layers.22.self_attn.v_proj.bias', 'text_model.encoder.layers.22.self_attn.v_proj.weight', 'text_model.encoder.layers.23.layer_norm1.bias', 'text_model.encoder.layers.23.layer_norm1.weight', 'text_model.encoder.layers.23.layer_norm2.bias', 'text_model.encoder.layers.23.layer_norm2.weight', 'text_model.encoder.layers.23.mlp.fc1.bias', 'text_model.encoder.layers.23.mlp.fc1.weight', 'text_model.encoder.layers.23.mlp.fc2.bias', 'text_model.encoder.layers.23.mlp.fc2.weight', 'text_model.encoder.layers.23.self_attn.k_proj.bias', 'text_model.encoder.layers.23.self_attn.k_proj.weight', 'text_model.encoder.layers.23.self_attn.out_proj.bias', 'text_model.encoder.layers.23.self_attn.out_proj.weight', 'text_model.encoder.layers.23.self_attn.q_proj.bias', 'text_model.encoder.layers.23.self_attn.q_proj.weight', 'text_model.encoder.layers.23.self_attn.v_proj.bias', 'text_model.encoder.layers.23.self_attn.v_proj.weight', 'text_model.encoder.layers.24.layer_norm1.bias', 'text_model.encoder.layers.24.layer_norm1.weight', 'text_model.encoder.layers.24.layer_norm2.bias', 'text_model.encoder.layers.24.layer_norm2.weight', 'text_model.encoder.layers.24.mlp.fc1.bias', 'text_model.encoder.layers.24.mlp.fc1.weight', 'text_model.encoder.layers.24.mlp.fc2.bias', 'text_model.encoder.layers.24.mlp.fc2.weight', 'text_model.encoder.layers.24.self_attn.k_proj.bias', 'text_model.encoder.layers.24.self_attn.k_proj.weight', 'text_model.encoder.layers.24.self_attn.out_proj.bias', 'text_model.encoder.layers.24.self_attn.out_proj.weight', 'text_model.encoder.layers.24.self_attn.q_proj.bias', 'text_model.encoder.layers.24.self_attn.q_proj.weight', 'text_model.encoder.layers.24.self_attn.v_proj.bias', 'text_model.encoder.layers.24.self_attn.v_proj.weight', 'text_model.encoder.layers.25.layer_norm1.bias', 'text_model.encoder.layers.25.layer_norm1.weight', 'text_model.encoder.layers.25.layer_norm2.bias', 'text_model.encoder.layers.25.layer_norm2.weight', 'text_model.encoder.layers.25.mlp.fc1.bias', 'text_model.encoder.layers.25.mlp.fc1.weight', 'text_model.encoder.layers.25.mlp.fc2.bias', 'text_model.encoder.layers.25.mlp.fc2.weight', 'text_model.encoder.layers.25.self_attn.k_proj.bias', 'text_model.encoder.layers.25.self_attn.k_proj.weight', 'text_model.encoder.layers.25.self_attn.out_proj.bias', 'text_model.encoder.layers.25.self_attn.out_proj.weight', 'text_model.encoder.layers.25.self_attn.q_proj.bias', 'text_model.encoder.layers.25.self_attn.q_proj.weight', 'text_model.encoder.layers.25.self_attn.v_proj.bias', 'text_model.encoder.layers.25.self_attn.v_proj.weight', 'text_model.encoder.layers.26.layer_norm1.bias', 'text_model.encoder.layers.26.layer_norm1.weight', 'text_model.encoder.layers.26.layer_norm2.bias', 'text_model.encoder.layers.26.layer_norm2.weight', 'text_model.encoder.layers.26.mlp.fc1.bias', 'text_model.encoder.layers.26.mlp.fc1.weight', 'text_model.encoder.layers.26.mlp.fc2.bias', 'text_model.encoder.layers.26.mlp.fc2.weight', 'text_model.encoder.layers.26.self_attn.k_proj.bias', 'text_model.encoder.layers.26.self_attn.k_proj.weight', 'text_model.encoder.layers.26.self_attn.out_proj.bias', 'text_model.encoder.layers.26.self_attn.out_proj.weight', 'text_model.encoder.layers.26.self_attn.q_proj.bias', 'text_model.encoder.layers.26.self_attn.q_proj.weight', 'text_model.encoder.layers.26.self_attn.v_proj.bias', 'text_model.encoder.layers.26.self_attn.v_proj.weight', 'text_model.encoder.layers.3.layer_norm1.bias', 'text_model.encoder.layers.3.layer_norm1.weight', 'text_model.encoder.layers.3.layer_norm2.bias', 'text_model.encoder.layers.3.layer_norm2.weight', 'text_model.encoder.layers.3.mlp.fc1.bias', 'text_model.encoder.layers.3.mlp.fc1.weight', 'text_model.encoder.layers.3.mlp.fc2.bias', 'text_model.encoder.layers.3.mlp.fc2.weight', 'text_model.encoder.layers.3.self_attn.k_proj.bias', 'text_model.encoder.layers.3.self_attn.k_proj.weight', 'text_model.encoder.layers.3.self_attn.out_proj.bias', 'text_model.encoder.layers.3.self_attn.out_proj.weight', 'text_model.encoder.layers.3.self_attn.q_proj.bias', 'text_model.encoder.layers.3.self_attn.q_proj.weight', 'text_model.encoder.layers.3.self_attn.v_proj.bias', 'text_model.encoder.layers.3.self_attn.v_proj.weight', 'text_model.encoder.layers.4.layer_norm1.bias', 'text_model.encoder.layers.4.layer_norm1.weight', 'text_model.encoder.layers.4.layer_norm2.bias', 'text_model.encoder.layers.4.layer_norm2.weight', 'text_model.encoder.layers.4.mlp.fc1.bias', 'text_model.encoder.layers.4.mlp.fc1.weight', 'text_model.encoder.layers.4.mlp.fc2.bias', 'text_model.encoder.layers.4.mlp.fc2.weight', 'text_model.encoder.layers.4.self_attn.k_proj.bias', 'text_model.encoder.layers.4.self_attn.k_proj.weight', 'text_model.encoder.layers.4.self_attn.out_proj.bias', 'text_model.encoder.layers.4.self_attn.out_proj.weight', 'text_model.encoder.layers.4.self_attn.q_proj.bias', 'text_model.encoder.layers.4.self_attn.q_proj.weight', 'text_model.encoder.layers.4.self_attn.v_proj.bias', 'text_model.encoder.layers.4.self_attn.v_proj.weight', 'text_model.encoder.layers.5.layer_norm1.bias', 'text_model.encoder.layers.5.layer_norm1.weight', 'text_model.encoder.layers.5.layer_norm2.bias', 'text_model.encoder.layers.5.layer_norm2.weight', 'text_model.encoder.layers.5.mlp.fc1.bias', 'text_model.encoder.layers.5.mlp.fc1.weight', 'text_model.encoder.layers.5.mlp.fc2.bias', 'text_model.encoder.layers.5.mlp.fc2.weight', 'text_model.encoder.layers.5.self_attn.k_proj.bias', 'text_model.encoder.layers.5.self_attn.k_proj.weight', 'text_model.encoder.layers.5.self_attn.out_proj.bias', 'text_model.encoder.layers.5.self_attn.out_proj.weight', 'text_model.encoder.layers.5.self_attn.q_proj.bias', 'text_model.encoder.layers.5.self_attn.q_proj.weight', 'text_model.encoder.layers.5.self_attn.v_proj.bias', 'text_model.encoder.layers.5.self_attn.v_proj.weight', 'text_model.encoder.layers.6.layer_norm1.bias', 'text_model.encoder.layers.6.layer_norm1.weight', 'text_model.encoder.layers.6.layer_norm2.bias', 'text_model.encoder.layers.6.layer_norm2.weight', 'text_model.encoder.layers.6.mlp.fc1.bias', 'text_model.encoder.layers.6.mlp.fc1.weight', 'text_model.encoder.layers.6.mlp.fc2.bias', 'text_model.encoder.layers.6.mlp.fc2.weight', 'text_model.encoder.layers.6.self_attn.k_proj.bias', 'text_model.encoder.layers.6.self_attn.k_proj.weight', 'text_model.encoder.layers.6.self_attn.out_proj.bias', 'text_model.encoder.layers.6.self_attn.out_proj.weight', 'text_model.encoder.layers.6.self_attn.q_proj.bias', 'text_model.encoder.layers.6.self_attn.q_proj.weight', 'text_model.encoder.layers.6.self_attn.v_proj.bias', 'text_model.encoder.layers.6.self_attn.v_proj.weight', 'text_model.encoder.layers.7.layer_norm1.bias', 'text_model.encoder.layers.7.layer_norm1.weight', 'text_model.encoder.layers.7.layer_norm2.bias', 'text_model.encoder.layers.7.layer_norm2.weight', 'text_model.encoder.layers.7.mlp.fc1.bias', 'text_model.encoder.layers.7.mlp.fc1.weight', 'text_model.encoder.layers.7.mlp.fc2.bias', 'text_model.encoder.layers.7.mlp.fc2.weight', 'text_model.encoder.layers.7.self_attn.k_proj.bias', 'text_model.encoder.layers.7.self_attn.k_proj.weight', 'text_model.encoder.layers.7.self_attn.out_proj.bias', 'text_model.encoder.layers.7.self_attn.out_proj.weight', 'text_model.encoder.layers.7.self_attn.q_proj.bias', 'text_model.encoder.layers.7.self_attn.q_proj.weight', 'text_model.encoder.layers.7.self_attn.v_proj.bias', 'text_model.encoder.layers.7.self_attn.v_proj.weight', 'text_model.encoder.layers.8.layer_norm1.bias', 'text_model.encoder.layers.8.layer_norm1.weight', 'text_model.encoder.layers.8.layer_norm2.bias', 'text_model.encoder.layers.8.layer_norm2.weight', 'text_model.encoder.layers.8.mlp.fc1.bias', 'text_model.encoder.layers.8.mlp.fc1.weight', 'text_model.encoder.layers.8.mlp.fc2.bias', 'text_model.encoder.layers.8.mlp.fc2.weight', 'text_model.encoder.layers.8.self_attn.k_proj.bias', 'text_model.encoder.layers.8.self_attn.k_proj.weight', 'text_model.encoder.layers.8.self_attn.out_proj.bias', 'text_model.encoder.layers.8.self_attn.out_proj.weight', 'text_model.encoder.layers.8.self_attn.q_proj.bias', 'text_model.encoder.layers.8.self_attn.q_proj.weight', 'text_model.encoder.layers.8.self_attn.v_proj.bias', 'text_model.encoder.layers.8.self_attn.v_proj.weight', 'text_model.encoder.layers.9.layer_norm1.bias', 'text_model.encoder.layers.9.layer_norm1.weight', 'text_model.encoder.layers.9.layer_norm2.bias', 'text_model.encoder.layers.9.layer_norm2.weight', 'text_model.encoder.layers.9.mlp.fc1.bias', 'text_model.encoder.layers.9.mlp.fc1.weight', 'text_model.encoder.layers.9.mlp.fc2.bias', 'text_model.encoder.layers.9.mlp.fc2.weight', 'text_model.encoder.layers.9.self_attn.k_proj.bias', 'text_model.encoder.layers.9.self_attn.k_proj.weight', 'text_model.encoder.layers.9.self_attn.out_proj.bias', 'text_model.encoder.layers.9.self_attn.out_proj.weight', 'text_model.encoder.layers.9.self_attn.q_proj.bias', 'text_model.encoder.layers.9.self_attn.q_proj.weight', 'text_model.encoder.layers.9.self_attn.v_proj.bias', 'text_model.encoder.layers.9.self_attn.v_proj.weight', 'text_model.final_layer_norm.bias', 'text_model.final_layer_norm.weight', 'text_model.head.bias', 'text_model.head.weight'] - This IS expected if you are initializing SiglipVisionModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model). - This IS NOT expected if you are initializing SiglipVisionModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model). 2025-02-14 05:03:01,242 - modeling_utils.py:4458 - _load_pretrained_model - INFO - All the weights of SiglipVisionModel were initialized from the model checkpoint at google/siglip-so400m-patch14-384. If your task is similar to the task the model of the checkpoint was trained on, you can already use SiglipVisionModel for predictions without further training. 2025-02-14 05:03:01,421 - image_processing_base.py:375 - get_image_processor_dict - INFO - loading configuration file preprocessor_config.json from cache at /root/.cache/huggingface/hub/models--google--siglip-so400m-patch14-384/snapshots/9fdffc58afc957d1a03a25b10dba0329ab15c2a3/preprocessor_config.json 2025-02-14 05:03:01,422 - image_processing_base.py:429 - from_dict - INFO - Image processor SiglipImageProcessor { "do_convert_rgb": null, "do_normalize": true, "do_rescale": true, "do_resize": true, "image_mean": [ 0.5, 0.5, 0.5 ], "image_processor_type": "SiglipImageProcessor", "image_std": [ 0.5, 0.5, 0.5 ], "processor_class": "SiglipProcessor", "resample": 3, "rescale_factor": 0.00392156862745098, "size": { "height": 384, "width": 384 } } 2025-02-14 05:03:02,082 - configuration_utils.py:733 - _get_config_dict - INFO - loading configuration file config.json from cache at /root/.cache/huggingface/hub/models--facebook--dinov2-giant/snapshots/611a9d42f2335e0f921f1e313ad3c1b7178d206d/config.json 2025-02-14 05:03:02,086 - configuration_utils.py:800 - from_dict - INFO - Model config Dinov2Config { "apply_layernorm": true, "architectures": [ "Dinov2Model" ], "attention_probs_dropout_prob": 0.0, "drop_path_rate": 0.0, "hidden_act": "gelu", "hidden_dropout_prob": 0.0, "hidden_size": 1536, "image_size": 518, "initializer_range": 0.02, "layer_norm_eps": 1e-06, "layerscale_value": 1.0, "mlp_ratio": 4, "model_type": "dinov2", "num_attention_heads": 24, "num_channels": 3, "num_hidden_layers": 40, "out_features": [ "stage40" ], "out_indices": [ 40 ], "patch_size": 14, "qkv_bias": true, "reshape_hidden_states": true, "stage_names": [ "stem", "stage1", "stage2", "stage3", "stage4", "stage5", "stage6", "stage7", "stage8", "stage9", "stage10", "stage11", "stage12", "stage13", "stage14", "stage15", "stage16", "stage17", "stage18", "stage19", "stage20", "stage21", "stage22", "stage23", "stage24", "stage25", "stage26", "stage27", "stage28", "stage29", "stage30", "stage31", "stage32", "stage33", "stage34", "stage35", "stage36", "stage37", "stage38", "stage39", "stage40" ], "torch_dtype": "float32", "transformers_version": "4.43.1", "use_swiglu_ffn": true } 2025-02-14 05:03:02,087 - modeling_utils.py:3621 - from_pretrained - INFO - loading weights file model.safetensors from cache at /root/.cache/huggingface/hub/models--facebook--dinov2-giant/snapshots/611a9d42f2335e0f921f1e313ad3c1b7178d206d/model.safetensors 2025-02-14 05:03:02,420 - modeling_utils.py:4450 - _load_pretrained_model - INFO - All model checkpoint weights were used when initializing Dinov2Model. 2025-02-14 05:03:02,421 - modeling_utils.py:4458 - _load_pretrained_model - INFO - All the weights of Dinov2Model were initialized from the model checkpoint at facebook/dinov2-giant. If your task is similar to the task the model of the checkpoint was trained on, you can already use Dinov2Model for predictions without further training. 2025-02-14 05:03:02,605 - image_processing_base.py:375 - get_image_processor_dict - INFO - loading configuration file preprocessor_config.json from cache at /root/.cache/huggingface/hub/models--facebook--dinov2-giant/snapshots/611a9d42f2335e0f921f1e313ad3c1b7178d206d/preprocessor_config.json 2025-02-14 05:03:02,608 - image_processing_base.py:429 - from_dict - INFO - Image processor BitImageProcessor { "crop_size": { "height": 378, "width": 378 }, "do_center_crop": true, "do_convert_rgb": true, "do_normalize": true, "do_rescale": true, "do_resize": true, "image_mean": [ 0.485, 0.456, 0.406 ], "image_processor_type": "BitImageProcessor", "image_std": [ 0.229, 0.224, 0.225 ], "resample": 3, "rescale_factor": 0.00392156862745098, "size": { "shortest_edge": 378 } } 2025-02-14 05:03:03,273 - finetune_llama.py:1239 - train - INFO - Total params: 3264865280 2025-02-14 05:03:03,273 - finetune_llama.py:1240 - train - INFO - Trainable params: 12589056 2025-02-14 05:03:03,273 - finetune_llama.py:1241 - train - INFO - LM head params: 394002432 2025-02-14 05:03:06,089 - trainer_callback.py:423 - add_callback - WARNING - You are adding a to the callbacks of this Trainer, but there is already one. The currentlist of callbacks is :DefaultFlowCallback TensorBoardCallback 2025-02-14 05:03:06,089 - trainer.py:648 - __init__ - INFO - Using auto half precision backend 2025-02-14 05:03:06,648 - trainer.py:2134 - _inner_training_loop - INFO - ***** Running training ***** 2025-02-14 05:03:06,648 - trainer.py:2135 - _inner_training_loop - INFO - Num examples = 554 2025-02-14 05:03:06,648 - trainer.py:2136 - _inner_training_loop - INFO - Num Epochs = 2 2025-02-14 05:03:06,649 - trainer.py:2137 - _inner_training_loop - INFO - Instantaneous batch size per device = 1 2025-02-14 05:03:06,649 - trainer.py:2140 - _inner_training_loop - INFO - Total train batch size (w. parallel, distributed & accumulation) = 1 2025-02-14 05:03:06,649 - trainer.py:2141 - _inner_training_loop - INFO - Gradient Accumulation steps = 1 2025-02-14 05:03:06,649 - trainer.py:2142 - _inner_training_loop - INFO - Total optimization steps = 1,108 2025-02-14 05:03:06,651 - trainer.py:2143 - _inner_training_loop - INFO - Number of trainable parameters = 406,591,488 2025-02-14 05:03:28,905 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:03:28,905 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 05:03:28,939 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 05:03:28,944 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:03:28,944 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 213, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 05:03:28,945 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:03:28,945 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 213, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 05:03:32,402 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 05:03:32,402 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 05:03:32,402 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.45 seconds 2025-02-14 05:03:32,402 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:03:32,402 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 12760.04 MB 2025-02-14 05:03:32,402 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 13547.39 MB 2025-02-14 05:03:32,402 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 787.35 MB 2025-02-14 05:03:32,402 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 13220.45 MB 2025-02-14 05:03:32,402 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 14554.23 MB 2025-02-14 05:03:32,402 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1333.79 MB 2025-02-14 05:03:32,402 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22492.26 MB 2025-02-14 05:03:32,464 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 05:03:32,464 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 05:03:32,464 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 05:03:32,464 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:03:32,464 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13547.32 MB 2025-02-14 05:03:32,464 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 13890.88 MB 2025-02-14 05:03:32,464 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 343.55 MB 2025-02-14 05:03:32,464 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 14554.23 MB 2025-02-14 05:03:32,464 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17439.92 MB 2025-02-14 05:03:32,464 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2885.68 MB 2025-02-14 05:03:32,464 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16515.39 MB 2025-02-14 05:03:33,583 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 05:03:33,583 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 05:03:33,583 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.12 seconds 2025-02-14 05:03:33,583 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:03:33,583 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13890.88 MB 2025-02-14 05:03:33,583 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14169.57 MB 2025-02-14 05:03:33,583 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 278.69 MB 2025-02-14 05:03:33,583 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17439.92 MB 2025-02-14 05:03:33,583 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 15323.89 MB 2025-02-14 05:03:33,583 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2116.03 MB 2025-02-14 05:03:33,583 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18146.50 MB 2025-02-14 05:03:33,592 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 05:03:33,592 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 05:03:33,592 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 05:03:33,592 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:03:33,592 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14169.57 MB 2025-02-14 05:03:33,592 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15161.33 MB 2025-02-14 05:03:33,592 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 991.76 MB 2025-02-14 05:03:33,592 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 15323.89 MB 2025-02-14 05:03:33,592 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 16320.04 MB 2025-02-14 05:03:33,592 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 996.15 MB 2025-02-14 05:03:33,592 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15905.49 MB 2025-02-14 05:03:33,717 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 05:03:33,717 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 05:03:33,717 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 05:03:33,717 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:03:33,717 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15161.33 MB 2025-02-14 05:03:33,717 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16338.34 MB 2025-02-14 05:03:33,717 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1177.01 MB 2025-02-14 05:03:33,717 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 16320.04 MB 2025-02-14 05:03:33,717 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20176.70 MB 2025-02-14 05:03:33,717 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3856.66 MB 2025-02-14 05:03:33,717 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19249.05 MB 2025-02-14 05:03:33,718 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 05:03:33,718 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 05:03:33,718 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 05:03:33,718 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:03:33,718 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14169.57 MB 2025-02-14 05:03:33,718 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16338.34 MB 2025-02-14 05:03:33,718 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2168.77 MB 2025-02-14 05:03:33,718 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 15323.89 MB 2025-02-14 05:03:33,718 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20176.70 MB 2025-02-14 05:03:33,718 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4852.81 MB 2025-02-14 05:03:33,718 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19249.05 MB 2025-02-14 05:03:33,804 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 05:03:33,804 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 05:03:33,804 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 05:03:33,804 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:03:33,805 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17143.45 MB 2025-02-14 05:03:33,805 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17546.12 MB 2025-02-14 05:03:33,805 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 402.68 MB 2025-02-14 05:03:33,805 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20176.70 MB 2025-02-14 05:03:33,805 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20396.90 MB 2025-02-14 05:03:33,805 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 220.20 MB 2025-02-14 05:03:33,805 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17919.91 MB 2025-02-14 05:03:33,827 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 05:03:33,827 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 05:03:33,827 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:03:33,827 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:03:33,827 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17762.90 MB 2025-02-14 05:03:33,827 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17967.68 MB 2025-02-14 05:03:33,827 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 204.79 MB 2025-02-14 05:03:33,827 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20396.90 MB 2025-02-14 05:03:33,827 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20401.09 MB 2025-02-14 05:03:33,827 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-14 05:03:33,827 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18025.36 MB 2025-02-14 05:03:33,829 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 05:03:33,829 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 05:03:33,829 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.88 seconds 2025-02-14 05:03:33,829 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:03:33,829 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 12017.34 MB 2025-02-14 05:03:33,829 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18168.75 MB 2025-02-14 05:03:33,829 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6151.41 MB 2025-02-14 05:03:33,829 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 12475.96 MB 2025-02-14 05:03:33,829 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20401.09 MB 2025-02-14 05:03:33,829 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7925.14 MB 2025-02-14 05:03:33,829 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18168.75 MB 2025-02-14 05:03:33,862 - logging.py:328 - warning_once - WARNING - The attention layers in this model are transitioning from computing the RoPE embeddings internally through `position_ids` (2D tensor with the indexes of the tokens), to using externally computed `position_embeddings` (Tuple of tensors, containing cos and sin). In v4.45 `position_ids` will be removed and `position_embeddings` will be mandatory. 2025-02-14 05:03:34,127 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 05:03:34,127 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 05:03:34,127 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.30 seconds 2025-02-14 05:03:34,127 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:03:34,127 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13144.06 MB 2025-02-14 05:03:34,127 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16158.88 MB 2025-02-14 05:03:34,127 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.82 MB 2025-02-14 05:03:34,127 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20401.09 MB 2025-02-14 05:03:34,127 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20401.09 MB 2025-02-14 05:03:34,127 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:03:34,127 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16460.25 MB 2025-02-14 05:03:34,145 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 05:03:34,148 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 05:03:34,155 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 05:03:34,155 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 05:03:34,155 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 05:03:34,156 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:03:34,156 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16158.88 MB 2025-02-14 05:03:34,156 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24597.90 MB 2025-02-14 05:03:34,156 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 05:03:34,156 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20401.09 MB 2025-02-14 05:03:34,156 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30891.05 MB 2025-02-14 05:03:34,156 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 05:03:34,156 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24597.90 MB 2025-02-14 05:03:34,319 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 05:03:34,321 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:03:34,321 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 05:03:34,322 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:03:34,322 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 05:03:34,326 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 05:03:34,327 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:03:34,327 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 05:03:34,327 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 05:04:42,714 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:04:42,715 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 05:04:42,720 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 05:04:42,725 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:04:42,725 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 290, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 05:04:42,726 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:04:42,726 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 290, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 05:04:47,141 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 05:04:47,141 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 05:04:47,141 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.41 seconds 2025-02-14 05:04:47,141 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:04:47,141 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14989.47 MB 2025-02-14 05:04:47,141 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16015.77 MB 2025-02-14 05:04:47,141 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1026.29 MB 2025-02-14 05:04:47,141 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43476.06 MB 2025-02-14 05:04:47,141 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18679.33 MB 2025-02-14 05:04:47,141 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24796.73 MB 2025-02-14 05:04:47,141 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24914.64 MB 2025-02-14 05:04:47,160 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 05:04:47,160 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 05:04:47,160 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:04:47,160 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:04:47,160 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16015.77 MB 2025-02-14 05:04:47,160 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16400.57 MB 2025-02-14 05:04:47,160 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 384.80 MB 2025-02-14 05:04:47,160 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18679.33 MB 2025-02-14 05:04:47,160 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22051.55 MB 2025-02-14 05:04:47,160 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3372.22 MB 2025-02-14 05:04:47,160 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19871.46 MB 2025-02-14 05:04:48,463 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 05:04:48,463 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 05:04:48,463 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.30 seconds 2025-02-14 05:04:48,463 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:04:48,463 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16400.57 MB 2025-02-14 05:04:48,463 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16764.20 MB 2025-02-14 05:04:48,463 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 363.63 MB 2025-02-14 05:04:48,463 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22051.55 MB 2025-02-14 05:04:48,463 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19581.11 MB 2025-02-14 05:04:48,463 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2470.45 MB 2025-02-14 05:04:48,463 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20741.91 MB 2025-02-14 05:04:48,473 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 05:04:48,473 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 05:04:48,473 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 05:04:48,473 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:04:48,473 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16764.20 MB 2025-02-14 05:04:48,473 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18058.23 MB 2025-02-14 05:04:48,473 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1294.03 MB 2025-02-14 05:04:48,473 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19581.11 MB 2025-02-14 05:04:48,473 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20877.15 MB 2025-02-14 05:04:48,473 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1296.04 MB 2025-02-14 05:04:48,473 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19029.17 MB 2025-02-14 05:04:48,617 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 05:04:48,617 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 05:04:48,617 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-14 05:04:48,617 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:04:48,617 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18058.23 MB 2025-02-14 05:04:48,617 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19593.92 MB 2025-02-14 05:04:48,617 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1535.69 MB 2025-02-14 05:04:48,617 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20877.15 MB 2025-02-14 05:04:48,617 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25092.42 MB 2025-02-14 05:04:48,617 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4215.28 MB 2025-02-14 05:04:48,617 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23391.73 MB 2025-02-14 05:04:48,618 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 05:04:48,618 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 05:04:48,618 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 05:04:48,618 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:04:48,618 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16764.20 MB 2025-02-14 05:04:48,618 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19593.92 MB 2025-02-14 05:04:48,618 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2829.73 MB 2025-02-14 05:04:48,618 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19581.11 MB 2025-02-14 05:04:48,618 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25092.42 MB 2025-02-14 05:04:48,618 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5511.32 MB 2025-02-14 05:04:48,618 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23391.73 MB 2025-02-14 05:04:48,730 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 05:04:48,730 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 05:04:48,730 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 05:04:48,730 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:04:48,730 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20644.40 MB 2025-02-14 05:04:48,730 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21169.79 MB 2025-02-14 05:04:48,730 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 525.40 MB 2025-02-14 05:04:48,730 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25092.42 MB 2025-02-14 05:04:48,730 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25377.64 MB 2025-02-14 05:04:48,730 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 285.21 MB 2025-02-14 05:04:48,730 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21654.63 MB 2025-02-14 05:04:48,744 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 05:04:48,744 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 05:04:48,744 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 05:04:48,744 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:04:48,744 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21452.63 MB 2025-02-14 05:04:48,744 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21658.67 MB 2025-02-14 05:04:48,744 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.04 MB 2025-02-14 05:04:48,744 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25377.64 MB 2025-02-14 05:04:48,744 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25381.83 MB 2025-02-14 05:04:48,744 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-14 05:04:48,744 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21754.19 MB 2025-02-14 05:04:48,745 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 05:04:48,745 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 05:04:48,745 - resource_logging.py:150 - __exit__ - DEBUG - Time: 6.02 seconds 2025-02-14 05:04:48,745 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:04:48,745 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13979.09 MB 2025-02-14 05:04:48,745 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21859.74 MB 2025-02-14 05:04:48,745 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7880.65 MB 2025-02-14 05:04:48,745 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43476.06 MB 2025-02-14 05:04:48,745 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25381.83 MB 2025-02-14 05:04:48,745 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18094.23 MB 2025-02-14 05:04:48,745 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21859.74 MB 2025-02-14 05:04:49,009 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 05:04:49,009 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 05:04:49,009 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 05:04:49,009 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:04:49,009 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21859.74 MB 2025-02-14 05:04:49,009 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24873.78 MB 2025-02-14 05:04:49,009 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-14 05:04:49,009 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25381.83 MB 2025-02-14 05:04:49,009 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26321.35 MB 2025-02-14 05:04:49,009 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 939.52 MB 2025-02-14 05:04:49,009 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25175.41 MB 2025-02-14 05:04:49,027 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 05:04:49,027 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 05:04:49,033 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 05:04:49,033 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 05:04:49,033 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:04:49,033 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:04:49,033 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18388.32 MB 2025-02-14 05:04:49,033 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26827.35 MB 2025-02-14 05:04:49,033 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 05:04:49,033 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26321.35 MB 2025-02-14 05:04:49,033 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36811.31 MB 2025-02-14 05:04:49,033 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 05:04:49,033 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26827.35 MB 2025-02-14 05:04:49,197 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 05:04:49,198 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:04:49,198 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 05:04:49,200 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:04:49,200 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 05:04:49,205 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 05:04:49,206 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:04:49,207 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 05:04:49,207 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 05:06:10,867 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:06:10,868 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 05:06:10,873 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 05:06:10,877 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:06:10,877 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1627, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 05:06:10,878 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:06:10,878 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1627, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 05:06:35,697 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 05:06:35,697 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 05:06:35,697 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.81 seconds 2025-02-14 05:06:35,697 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:06:35,697 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24305.90 MB 2025-02-14 05:06:35,697 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30064.68 MB 2025-02-14 05:06:35,697 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5758.78 MB 2025-02-14 05:06:35,697 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49396.32 MB 2025-02-14 05:06:35,697 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39447.43 MB 2025-02-14 05:06:35,697 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9948.89 MB 2025-02-14 05:06:35,697 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38987.41 MB 2025-02-14 05:06:35,776 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 05:06:35,776 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 05:06:35,776 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 05:06:35,776 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:06:35,776 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30064.68 MB 2025-02-14 05:06:35,776 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24236.12 MB 2025-02-14 05:06:35,776 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5828.57 MB 2025-02-14 05:06:35,776 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39447.43 MB 2025-02-14 05:06:35,776 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48897.20 MB 2025-02-14 05:06:35,776 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9449.77 MB 2025-02-14 05:06:35,776 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40945.26 MB 2025-02-14 05:06:37,684 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 05:06:37,684 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 05:06:37,684 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 05:06:37,684 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:06:37,684 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24236.12 MB 2025-02-14 05:06:37,684 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24766.96 MB 2025-02-14 05:06:37,684 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 05:06:37,684 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48897.20 MB 2025-02-14 05:06:37,684 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30909.92 MB 2025-02-14 05:06:37,684 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17987.27 MB 2025-02-14 05:06:37,684 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28746.29 MB 2025-02-14 05:06:37,700 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 05:06:37,700 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 05:06:37,700 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 05:06:37,700 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:06:37,700 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24766.96 MB 2025-02-14 05:06:37,700 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26656.49 MB 2025-02-14 05:06:37,700 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 05:06:37,700 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30909.92 MB 2025-02-14 05:06:37,700 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31853.64 MB 2025-02-14 05:06:37,700 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 05:06:37,700 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28073.92 MB 2025-02-14 05:06:37,912 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 05:06:37,912 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 05:06:37,913 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 05:06:37,913 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:06:37,913 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26656.49 MB 2025-02-14 05:06:37,913 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28898.35 MB 2025-02-14 05:06:37,913 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 05:06:37,913 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31853.64 MB 2025-02-14 05:06:37,913 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37515.95 MB 2025-02-14 05:06:37,913 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 05:06:37,913 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34442.63 MB 2025-02-14 05:06:37,913 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 05:06:37,913 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 05:06:37,913 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 05:06:37,913 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:06:37,913 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24766.96 MB 2025-02-14 05:06:37,913 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28898.35 MB 2025-02-14 05:06:37,913 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 05:06:37,913 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30909.92 MB 2025-02-14 05:06:37,913 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37515.95 MB 2025-02-14 05:06:37,913 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 05:06:37,913 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34442.63 MB 2025-02-14 05:06:38,083 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 05:06:38,083 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 05:06:38,083 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 05:06:38,083 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:06:38,083 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30431.89 MB 2025-02-14 05:06:38,083 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31198.89 MB 2025-02-14 05:06:38,083 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 05:06:38,083 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37515.95 MB 2025-02-14 05:06:38,083 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37931.19 MB 2025-02-14 05:06:38,083 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 05:06:38,083 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31906.68 MB 2025-02-14 05:06:38,102 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 05:06:38,102 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 05:06:38,102 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:06:38,102 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:06:38,102 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31611.78 MB 2025-02-14 05:06:38,102 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31843.21 MB 2025-02-14 05:06:38,102 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 231.43 MB 2025-02-14 05:06:38,102 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37931.19 MB 2025-02-14 05:06:38,102 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37931.19 MB 2025-02-14 05:06:38,102 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:06:38,102 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32050.89 MB 2025-02-14 05:06:38,103 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 05:06:38,103 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 05:06:38,103 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.22 seconds 2025-02-14 05:06:38,103 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:06:38,103 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18637.30 MB 2025-02-14 05:06:38,103 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32044.28 MB 2025-02-14 05:06:38,103 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13406.98 MB 2025-02-14 05:06:38,103 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49396.32 MB 2025-02-14 05:06:38,103 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37931.19 MB 2025-02-14 05:06:38,103 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11465.13 MB 2025-02-14 05:06:38,103 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32050.89 MB 2025-02-14 05:06:38,370 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 05:06:38,370 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 05:06:38,370 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 05:06:38,370 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:06:38,370 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32044.28 MB 2025-02-14 05:06:38,370 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23641.69 MB 2025-02-14 05:06:38,370 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8402.59 MB 2025-02-14 05:06:38,370 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37931.19 MB 2025-02-14 05:06:38,370 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37931.19 MB 2025-02-14 05:06:38,370 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:06:38,370 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34555.95 MB 2025-02-14 05:06:38,387 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 05:06:38,388 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 05:06:38,394 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 05:06:38,394 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 05:06:38,394 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:06:38,394 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:06:38,394 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23641.69 MB 2025-02-14 05:06:38,394 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32080.38 MB 2025-02-14 05:06:38,394 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8438.69 MB 2025-02-14 05:06:38,394 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37931.19 MB 2025-02-14 05:06:38,394 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42127.59 MB 2025-02-14 05:06:38,394 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4196.40 MB 2025-02-14 05:06:38,394 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32080.38 MB 2025-02-14 05:06:38,565 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 05:06:38,567 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:06:38,567 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 05:06:38,568 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:06:38,568 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 05:06:38,573 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 05:06:38,574 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:06:38,574 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 05:06:38,574 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 05:06:53,297 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:06:53,297 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 05:06:53,302 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 05:06:53,305 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:06:53,305 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1754, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 05:06:53,306 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:06:53,306 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1754, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 05:07:20,404 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 05:07:20,404 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 05:07:20,404 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.09 seconds 2025-02-14 05:07:20,404 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:07:20,404 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25190.86 MB 2025-02-14 05:07:20,404 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31398.43 MB 2025-02-14 05:07:20,404 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6207.57 MB 2025-02-14 05:07:20,404 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50516.20 MB 2025-02-14 05:07:20,404 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39892.03 MB 2025-02-14 05:07:20,404 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10624.17 MB 2025-02-14 05:07:20,404 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40325.35 MB 2025-02-14 05:07:20,510 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 05:07:20,510 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 05:07:20,510 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 05:07:20,510 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:07:20,510 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31398.43 MB 2025-02-14 05:07:20,510 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24896.35 MB 2025-02-14 05:07:20,510 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6502.08 MB 2025-02-14 05:07:20,510 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39892.03 MB 2025-02-14 05:07:20,510 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 59068.38 MB 2025-02-14 05:07:20,510 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 19176.36 MB 2025-02-14 05:07:20,510 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49844.37 MB 2025-02-14 05:07:22,440 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 05:07:22,440 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 05:07:22,440 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 05:07:22,440 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:07:22,440 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24896.35 MB 2025-02-14 05:07:22,440 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25427.19 MB 2025-02-14 05:07:22,440 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 05:07:22,440 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59068.38 MB 2025-02-14 05:07:22,440 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30905.73 MB 2025-02-14 05:07:22,440 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28162.65 MB 2025-02-14 05:07:22,440 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29406.52 MB 2025-02-14 05:07:22,463 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 05:07:22,463 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 05:07:22,463 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:07:22,463 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:07:22,463 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25427.19 MB 2025-02-14 05:07:22,463 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27316.72 MB 2025-02-14 05:07:22,463 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 05:07:22,463 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30905.73 MB 2025-02-14 05:07:22,463 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31849.45 MB 2025-02-14 05:07:22,463 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 05:07:22,463 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28734.15 MB 2025-02-14 05:07:22,674 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 05:07:22,674 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 05:07:22,674 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 05:07:22,674 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:07:22,674 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27316.72 MB 2025-02-14 05:07:22,674 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29558.58 MB 2025-02-14 05:07:22,674 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 05:07:22,674 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31849.45 MB 2025-02-14 05:07:22,674 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37983.62 MB 2025-02-14 05:07:22,674 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 05:07:22,674 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35102.86 MB 2025-02-14 05:07:22,675 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 05:07:22,675 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 05:07:22,675 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 05:07:22,675 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:07:22,675 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25427.19 MB 2025-02-14 05:07:22,675 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29558.58 MB 2025-02-14 05:07:22,675 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 05:07:22,675 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30905.73 MB 2025-02-14 05:07:22,675 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37983.62 MB 2025-02-14 05:07:22,675 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7077.89 MB 2025-02-14 05:07:22,675 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35102.86 MB 2025-02-14 05:07:22,838 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 05:07:22,838 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 05:07:22,838 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 05:07:22,838 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:07:22,838 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31092.12 MB 2025-02-14 05:07:22,838 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31859.12 MB 2025-02-14 05:07:22,838 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 05:07:22,838 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37983.62 MB 2025-02-14 05:07:22,838 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38400.95 MB 2025-02-14 05:07:22,838 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 05:07:22,838 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32566.91 MB 2025-02-14 05:07:22,857 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 05:07:22,857 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 05:07:22,857 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:07:22,857 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:07:22,857 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32272.01 MB 2025-02-14 05:07:22,857 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32500.57 MB 2025-02-14 05:07:22,857 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.56 MB 2025-02-14 05:07:22,857 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38400.95 MB 2025-02-14 05:07:22,857 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38400.95 MB 2025-02-14 05:07:22,857 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:07:22,857 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32740.12 MB 2025-02-14 05:07:22,858 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 05:07:22,858 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 05:07:22,858 - resource_logging.py:150 - __exit__ - DEBUG - Time: 29.55 seconds 2025-02-14 05:07:22,858 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:07:22,858 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19079.78 MB 2025-02-14 05:07:22,858 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32700.88 MB 2025-02-14 05:07:22,858 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13621.10 MB 2025-02-14 05:07:22,858 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50516.20 MB 2025-02-14 05:07:22,858 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38400.95 MB 2025-02-14 05:07:22,858 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12115.25 MB 2025-02-14 05:07:22,858 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32740.12 MB 2025-02-14 05:07:23,127 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 05:07:23,127 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 05:07:23,127 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 05:07:23,127 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:07:23,127 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32700.88 MB 2025-02-14 05:07:23,127 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24072.36 MB 2025-02-14 05:07:23,127 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8628.52 MB 2025-02-14 05:07:23,127 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38400.95 MB 2025-02-14 05:07:23,127 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38400.95 MB 2025-02-14 05:07:23,127 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:07:23,127 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35203.03 MB 2025-02-14 05:07:23,145 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8131, cut from 8133 2025-02-14 05:07:23,145 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 05:07:23,151 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 05:07:23,151 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 05:07:23,151 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:07:23,151 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:07:23,151 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24072.36 MB 2025-02-14 05:07:23,151 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32480.10 MB 2025-02-14 05:07:23,151 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8407.74 MB 2025-02-14 05:07:23,151 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38400.95 MB 2025-02-14 05:07:23,151 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46760.20 MB 2025-02-14 05:07:23,151 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8359.25 MB 2025-02-14 05:07:23,151 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32480.10 MB 2025-02-14 05:07:23,313 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7923] 2025-02-14 05:07:23,314 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:07:23,314 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 05:07:23,315 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:07:23,315 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 05:07:23,320 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 05:07:23,321 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:07:23,321 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 05:07:23,321 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 05:07:33,459 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:07:33,459 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 05:07:33,464 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 05:07:33,468 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:07:33,468 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 321, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 05:07:33,469 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:07:33,469 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 321, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 05:07:38,504 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 05:07:38,504 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 05:07:38,504 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.03 seconds 2025-02-14 05:07:38,504 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:07:38,504 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15205.49 MB 2025-02-14 05:07:38,504 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16341.49 MB 2025-02-14 05:07:38,504 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1136.00 MB 2025-02-14 05:07:38,504 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55119.45 MB 2025-02-14 05:07:38,504 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22045.26 MB 2025-02-14 05:07:38,504 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33074.18 MB 2025-02-14 05:07:38,504 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25356.33 MB 2025-02-14 05:07:38,525 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 05:07:38,525 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 05:07:38,525 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:07:38,525 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:07:38,525 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16341.49 MB 2025-02-14 05:07:38,525 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16660.05 MB 2025-02-14 05:07:38,525 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 318.56 MB 2025-02-14 05:07:38,525 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22045.26 MB 2025-02-14 05:07:38,525 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23064.48 MB 2025-02-14 05:07:38,525 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1019.22 MB 2025-02-14 05:07:38,525 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20390.29 MB 2025-02-14 05:07:39,908 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 05:07:39,908 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 05:07:39,908 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.38 seconds 2025-02-14 05:07:39,908 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:07:39,908 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16660.05 MB 2025-02-14 05:07:39,908 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17042.26 MB 2025-02-14 05:07:39,908 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 382.21 MB 2025-02-14 05:07:39,908 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23064.48 MB 2025-02-14 05:07:39,908 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23064.48 MB 2025-02-14 05:07:39,908 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:07:39,908 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21000.36 MB 2025-02-14 05:07:39,919 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 05:07:39,919 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 05:07:39,919 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 05:07:39,919 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:07:39,919 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17042.26 MB 2025-02-14 05:07:39,919 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18403.31 MB 2025-02-14 05:07:39,919 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1361.05 MB 2025-02-14 05:07:39,919 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23064.48 MB 2025-02-14 05:07:39,919 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23064.48 MB 2025-02-14 05:07:39,919 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:07:39,919 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19423.86 MB 2025-02-14 05:07:40,074 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 05:07:40,075 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 05:07:40,075 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 05:07:40,075 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:07:40,075 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18403.31 MB 2025-02-14 05:07:40,075 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20017.46 MB 2025-02-14 05:07:40,075 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1614.15 MB 2025-02-14 05:07:40,075 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23064.48 MB 2025-02-14 05:07:40,075 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26122.13 MB 2025-02-14 05:07:40,075 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3057.65 MB 2025-02-14 05:07:40,075 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24009.33 MB 2025-02-14 05:07:40,075 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 05:07:40,075 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 05:07:40,075 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 05:07:40,075 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:07:40,075 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17042.26 MB 2025-02-14 05:07:40,075 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20017.46 MB 2025-02-14 05:07:40,075 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2975.21 MB 2025-02-14 05:07:40,075 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23064.48 MB 2025-02-14 05:07:40,075 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26122.13 MB 2025-02-14 05:07:40,075 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3057.65 MB 2025-02-14 05:07:40,075 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24009.33 MB 2025-02-14 05:07:40,198 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 05:07:40,198 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 05:07:40,198 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 05:07:40,198 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:07:40,198 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21121.61 MB 2025-02-14 05:07:40,198 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21673.85 MB 2025-02-14 05:07:40,198 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 552.24 MB 2025-02-14 05:07:40,198 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26122.13 MB 2025-02-14 05:07:40,198 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26422.02 MB 2025-02-14 05:07:40,198 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 299.89 MB 2025-02-14 05:07:40,198 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22183.46 MB 2025-02-14 05:07:40,212 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 05:07:40,212 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 05:07:40,212 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 05:07:40,212 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:07:40,212 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21971.14 MB 2025-02-14 05:07:40,212 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22199.94 MB 2025-02-14 05:07:40,212 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.80 MB 2025-02-14 05:07:40,212 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26422.02 MB 2025-02-14 05:07:40,212 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26422.02 MB 2025-02-14 05:07:40,212 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:07:40,212 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22285.05 MB 2025-02-14 05:07:40,213 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 05:07:40,213 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 05:07:40,213 - resource_logging.py:150 - __exit__ - DEBUG - Time: 6.74 seconds 2025-02-14 05:07:40,213 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:07:40,213 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14087.10 MB 2025-02-14 05:07:40,213 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22400.67 MB 2025-02-14 05:07:40,213 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8313.57 MB 2025-02-14 05:07:40,213 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55119.45 MB 2025-02-14 05:07:40,213 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26422.02 MB 2025-02-14 05:07:40,213 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28697.43 MB 2025-02-14 05:07:40,213 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22400.67 MB 2025-02-14 05:07:40,495 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 05:07:40,495 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 05:07:40,495 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.28 seconds 2025-02-14 05:07:40,495 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:07:40,495 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22400.67 MB 2025-02-14 05:07:40,495 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25409.54 MB 2025-02-14 05:07:40,495 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3008.87 MB 2025-02-14 05:07:40,495 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26422.02 MB 2025-02-14 05:07:40,495 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26824.67 MB 2025-02-14 05:07:40,495 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 402.65 MB 2025-02-14 05:07:40,495 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25710.88 MB 2025-02-14 05:07:40,513 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8148, cut from 8150 2025-02-14 05:07:40,513 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 1,'] 2025-02-14 05:07:40,519 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 05:07:40,519 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 05:07:40,519 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:07:40,519 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:07:40,519 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18557.59 MB 2025-02-14 05:07:40,519 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26982.54 MB 2025-02-14 05:07:40,519 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8424.95 MB 2025-02-14 05:07:40,519 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26824.67 MB 2025-02-14 05:07:40,519 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37295.75 MB 2025-02-14 05:07:40,519 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10471.08 MB 2025-02-14 05:07:40,519 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26982.54 MB 2025-02-14 05:07:40,681 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7940] 2025-02-14 05:07:40,682 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:07:40,682 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 05:07:40,683 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:07:40,683 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 05:07:40,688 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 05:07:40,689 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:07:40,689 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 05:07:40,689 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 1,'] 2025-02-14 05:07:49,579 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:07:49,579 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 05:07:49,584 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 05:07:49,587 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:07:49,587 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 191, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 05:07:49,588 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:07:49,588 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 191, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 05:07:52,593 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 05:07:52,594 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 05:07:52,594 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.00 seconds 2025-02-14 05:07:52,594 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:07:52,594 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14299.63 MB 2025-02-14 05:07:52,594 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14975.56 MB 2025-02-14 05:07:52,594 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 675.94 MB 2025-02-14 05:07:52,594 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45671.78 MB 2025-02-14 05:07:52,594 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17926.46 MB 2025-02-14 05:07:52,594 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27745.32 MB 2025-02-14 05:07:52,594 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23853.72 MB 2025-02-14 05:07:52,607 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 05:07:52,607 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 05:07:52,607 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 05:07:52,607 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:07:52,607 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14975.56 MB 2025-02-14 05:07:52,607 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15303.05 MB 2025-02-14 05:07:52,607 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 327.49 MB 2025-02-14 05:07:52,607 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17926.46 MB 2025-02-14 05:07:52,607 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18603.84 MB 2025-02-14 05:07:52,607 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 677.38 MB 2025-02-14 05:07:52,607 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17711.51 MB 2025-02-14 05:07:53,528 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 05:07:53,528 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 05:07:53,528 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.92 seconds 2025-02-14 05:07:53,528 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:07:53,528 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15303.05 MB 2025-02-14 05:07:53,528 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15556.53 MB 2025-02-14 05:07:53,528 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 253.48 MB 2025-02-14 05:07:53,528 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18603.84 MB 2025-02-14 05:07:53,528 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18329.11 MB 2025-02-14 05:07:53,528 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -274.73 MB 2025-02-14 05:07:53,528 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19495.69 MB 2025-02-14 05:07:53,536 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 05:07:53,536 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 05:07:53,536 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 05:07:53,536 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:07:53,536 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15556.47 MB 2025-02-14 05:07:53,536 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16458.76 MB 2025-02-14 05:07:53,536 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 902.30 MB 2025-02-14 05:07:53,536 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18329.11 MB 2025-02-14 05:07:53,536 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18780.00 MB 2025-02-14 05:07:53,536 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 450.89 MB 2025-02-14 05:07:53,536 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17135.59 MB 2025-02-14 05:07:53,639 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 05:07:53,639 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 05:07:53,639 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 05:07:53,639 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:07:53,639 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16458.76 MB 2025-02-14 05:07:53,639 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17529.54 MB 2025-02-14 05:07:53,639 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1070.78 MB 2025-02-14 05:07:53,639 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18780.00 MB 2025-02-14 05:07:53,639 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21711.81 MB 2025-02-14 05:07:53,639 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2931.82 MB 2025-02-14 05:07:53,639 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20177.95 MB 2025-02-14 05:07:53,640 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 05:07:53,640 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 05:07:53,640 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 05:07:53,640 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:07:53,640 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15556.47 MB 2025-02-14 05:07:53,640 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17529.54 MB 2025-02-14 05:07:53,640 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1973.08 MB 2025-02-14 05:07:53,640 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18329.11 MB 2025-02-14 05:07:53,640 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21711.81 MB 2025-02-14 05:07:53,640 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3382.71 MB 2025-02-14 05:07:53,640 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20177.95 MB 2025-02-14 05:07:53,720 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 05:07:53,720 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 05:07:53,720 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 05:07:53,720 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:07:53,720 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18261.81 MB 2025-02-14 05:07:53,720 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18628.05 MB 2025-02-14 05:07:53,720 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 366.24 MB 2025-02-14 05:07:53,720 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21711.81 MB 2025-02-14 05:07:53,720 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21908.95 MB 2025-02-14 05:07:53,720 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 197.13 MB 2025-02-14 05:07:53,720 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18972.06 MB 2025-02-14 05:07:53,730 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 05:07:53,730 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 05:07:53,730 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 05:07:53,730 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:07:53,730 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18825.21 MB 2025-02-14 05:07:53,730 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19050.29 MB 2025-02-14 05:07:53,730 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 225.07 MB 2025-02-14 05:07:53,730 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21908.95 MB 2025-02-14 05:07:53,730 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21908.95 MB 2025-02-14 05:07:53,730 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:07:53,730 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19086.72 MB 2025-02-14 05:07:53,731 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 05:07:53,731 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 05:07:53,731 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.14 seconds 2025-02-14 05:07:53,731 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:07:53,731 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13634.17 MB 2025-02-14 05:07:53,731 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19251.29 MB 2025-02-14 05:07:53,731 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5617.12 MB 2025-02-14 05:07:53,732 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45671.78 MB 2025-02-14 05:07:53,732 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21908.95 MB 2025-02-14 05:07:53,732 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23762.83 MB 2025-02-14 05:07:53,732 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19251.29 MB 2025-02-14 05:07:53,999 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 05:07:53,999 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 05:07:53,999 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 05:07:53,999 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:07:53,999 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19251.29 MB 2025-02-14 05:07:53,999 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17650.90 MB 2025-02-14 05:07:53,999 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1600.39 MB 2025-02-14 05:07:54,000 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21908.95 MB 2025-02-14 05:07:54,000 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21908.95 MB 2025-02-14 05:07:54,000 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:07:54,000 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19251.30 MB 2025-02-14 05:07:54,018 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8159, cut from 8161 2025-02-14 05:07:54,018 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 05:07:54,024 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 05:07:54,024 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 05:07:54,024 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:07:54,024 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:07:54,024 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17650.90 MB 2025-02-14 05:07:54,024 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26086.49 MB 2025-02-14 05:07:54,024 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8435.59 MB 2025-02-14 05:07:54,024 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21908.95 MB 2025-02-14 05:07:54,024 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32394.71 MB 2025-02-14 05:07:54,024 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10485.76 MB 2025-02-14 05:07:54,024 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26086.49 MB 2025-02-14 05:07:54,186 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7951] 2025-02-14 05:07:54,187 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:07:54,187 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 05:07:54,188 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:07:54,188 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 05:07:54,193 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 05:07:54,194 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:07:54,194 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 05:07:54,194 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 05:08:52,517 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:08:52,517 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 05:08:52,522 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 05:08:52,526 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:08:52,526 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 170, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 05:08:52,527 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:08:52,527 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 170, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 05:08:55,158 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 05:08:55,158 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 05:08:55,158 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.63 seconds 2025-02-14 05:08:55,158 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:08:55,159 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14153.77 MB 2025-02-14 05:08:55,159 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14755.39 MB 2025-02-14 05:08:55,159 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 601.62 MB 2025-02-14 05:08:55,159 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40783.31 MB 2025-02-14 05:08:55,159 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18100.52 MB 2025-02-14 05:08:55,159 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22682.80 MB 2025-02-14 05:08:55,159 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23625.95 MB 2025-02-14 05:08:55,176 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 05:08:55,176 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 05:08:55,176 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 05:08:55,176 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:08:55,176 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14755.39 MB 2025-02-14 05:08:55,176 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14984.33 MB 2025-02-14 05:08:55,176 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.93 MB 2025-02-14 05:08:55,176 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18100.52 MB 2025-02-14 05:08:55,176 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18670.94 MB 2025-02-14 05:08:55,176 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 570.43 MB 2025-02-14 05:08:55,176 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17052.92 MB 2025-02-14 05:08:55,985 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 05:08:55,985 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 05:08:55,985 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.81 seconds 2025-02-14 05:08:55,985 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:08:55,985 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14984.33 MB 2025-02-14 05:08:55,985 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15197.99 MB 2025-02-14 05:08:55,985 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 213.66 MB 2025-02-14 05:08:55,985 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18670.94 MB 2025-02-14 05:08:55,985 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18331.21 MB 2025-02-14 05:08:55,985 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -339.74 MB 2025-02-14 05:08:55,985 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19155.80 MB 2025-02-14 05:08:55,997 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 05:08:55,997 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 05:08:55,997 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 05:08:55,997 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:08:55,997 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15197.92 MB 2025-02-14 05:08:55,997 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15958.28 MB 2025-02-14 05:08:55,997 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 760.35 MB 2025-02-14 05:08:55,997 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18331.21 MB 2025-02-14 05:08:55,997 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18331.21 MB 2025-02-14 05:08:55,997 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:08:55,997 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16528.80 MB 2025-02-14 05:08:56,118 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 05:08:56,118 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 05:08:56,118 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 05:08:56,118 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:08:56,118 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15958.28 MB 2025-02-14 05:08:56,118 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16860.66 MB 2025-02-14 05:08:56,118 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 902.39 MB 2025-02-14 05:08:56,118 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18331.21 MB 2025-02-14 05:08:56,118 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20239.61 MB 2025-02-14 05:08:56,118 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1908.41 MB 2025-02-14 05:08:56,118 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19092.20 MB 2025-02-14 05:08:56,120 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 05:08:56,120 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 05:08:56,120 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 05:08:56,120 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:08:56,120 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15197.92 MB 2025-02-14 05:08:56,120 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16860.66 MB 2025-02-14 05:08:56,120 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1662.74 MB 2025-02-14 05:08:56,120 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18331.21 MB 2025-02-14 05:08:56,120 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20239.61 MB 2025-02-14 05:08:56,120 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1908.41 MB 2025-02-14 05:08:56,120 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19092.20 MB 2025-02-14 05:08:56,235 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 05:08:56,235 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 05:08:56,235 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 05:08:56,235 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:08:56,235 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17477.91 MB 2025-02-14 05:08:56,235 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17786.63 MB 2025-02-14 05:08:56,235 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 308.72 MB 2025-02-14 05:08:56,235 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20239.61 MB 2025-02-14 05:08:56,235 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20405.29 MB 2025-02-14 05:08:56,235 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 165.68 MB 2025-02-14 05:08:56,235 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18078.93 MB 2025-02-14 05:08:56,250 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 05:08:56,250 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 05:08:56,250 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 05:08:56,250 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:08:56,251 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17952.83 MB 2025-02-14 05:08:56,251 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18180.93 MB 2025-02-14 05:08:56,251 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.11 MB 2025-02-14 05:08:56,251 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20405.29 MB 2025-02-14 05:08:56,251 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20405.29 MB 2025-02-14 05:08:56,251 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:08:56,251 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18199.08 MB 2025-02-14 05:08:56,253 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 05:08:56,253 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 05:08:56,253 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.72 seconds 2025-02-14 05:08:56,253 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:08:56,253 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13561.48 MB 2025-02-14 05:08:56,253 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18381.88 MB 2025-02-14 05:08:56,253 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4820.40 MB 2025-02-14 05:08:56,253 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40783.31 MB 2025-02-14 05:08:56,253 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20405.29 MB 2025-02-14 05:08:56,253 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20378.03 MB 2025-02-14 05:08:56,253 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18381.88 MB 2025-02-14 05:08:56,544 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 05:08:56,544 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 05:08:56,544 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-14 05:08:56,545 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:08:56,545 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18381.88 MB 2025-02-14 05:08:56,545 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17436.30 MB 2025-02-14 05:08:56,545 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -945.59 MB 2025-02-14 05:08:56,545 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20405.29 MB 2025-02-14 05:08:56,545 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20405.29 MB 2025-02-14 05:08:56,545 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:08:56,545 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19185.13 MB 2025-02-14 05:08:56,569 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8157, cut from 8159 2025-02-14 05:08:56,570 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 05:08:56,596 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 05:08:56,596 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 05:08:56,596 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-14 05:08:56,596 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:08:56,596 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17436.30 MB 2025-02-14 05:08:56,596 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25870.92 MB 2025-02-14 05:08:56,596 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8434.62 MB 2025-02-14 05:08:56,596 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20405.29 MB 2025-02-14 05:08:56,596 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30886.85 MB 2025-02-14 05:08:56,596 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10481.57 MB 2025-02-14 05:08:56,596 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25870.92 MB 2025-02-14 05:08:56,859 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7949] 2025-02-14 05:08:56,862 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:08:56,862 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 05:08:56,864 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:08:56,864 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 05:08:56,872 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 05:08:56,875 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:08:56,875 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 05:08:56,875 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 05:09:38,717 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:09:38,717 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 05:09:38,722 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 05:09:38,726 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:09:38,726 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1377, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 05:09:38,727 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:09:38,727 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1377, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 05:09:59,903 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 05:09:59,904 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 05:09:59,904 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.17 seconds 2025-02-14 05:09:59,904 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:09:59,904 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22563.86 MB 2025-02-14 05:09:59,904 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27437.64 MB 2025-02-14 05:09:59,904 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4873.78 MB 2025-02-14 05:09:59,904 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39271.27 MB 2025-02-14 05:09:59,904 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38551.95 MB 2025-02-14 05:09:59,904 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -719.32 MB 2025-02-14 05:09:59,904 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36338.59 MB 2025-02-14 05:09:59,984 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 05:09:59,984 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 05:09:59,984 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 05:09:59,984 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:09:59,984 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27437.64 MB 2025-02-14 05:09:59,984 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22936.44 MB 2025-02-14 05:09:59,984 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4501.20 MB 2025-02-14 05:09:59,984 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38551.95 MB 2025-02-14 05:09:59,984 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48175.78 MB 2025-02-14 05:09:59,984 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9623.83 MB 2025-02-14 05:09:59,984 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41911.52 MB 2025-02-14 05:10:01,900 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 05:10:01,900 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 05:10:01,900 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 05:10:01,900 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:10:01,900 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22936.44 MB 2025-02-14 05:10:01,900 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23467.29 MB 2025-02-14 05:10:01,900 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 05:10:01,900 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48175.78 MB 2025-02-14 05:10:01,900 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33678.16 MB 2025-02-14 05:10:01,900 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14497.61 MB 2025-02-14 05:10:01,900 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27446.95 MB 2025-02-14 05:10:01,913 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 05:10:01,913 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 05:10:01,913 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 05:10:01,913 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:10:01,913 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23467.29 MB 2025-02-14 05:10:01,913 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25356.82 MB 2025-02-14 05:10:01,913 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 05:10:01,913 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33678.16 MB 2025-02-14 05:10:01,913 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33678.16 MB 2025-02-14 05:10:01,913 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:10:01,913 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26774.25 MB 2025-02-14 05:10:02,125 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 05:10:02,126 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 05:10:02,126 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 05:10:02,126 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:10:02,126 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25356.82 MB 2025-02-14 05:10:02,126 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27598.68 MB 2025-02-14 05:10:02,126 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 05:10:02,126 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33678.16 MB 2025-02-14 05:10:02,126 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37453.04 MB 2025-02-14 05:10:02,126 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 05:10:02,126 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33142.96 MB 2025-02-14 05:10:02,126 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 05:10:02,126 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 05:10:02,126 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 05:10:02,126 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:10:02,126 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23467.29 MB 2025-02-14 05:10:02,126 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27598.68 MB 2025-02-14 05:10:02,126 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 05:10:02,126 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33678.16 MB 2025-02-14 05:10:02,126 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37453.04 MB 2025-02-14 05:10:02,126 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 05:10:02,126 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33142.96 MB 2025-02-14 05:10:02,291 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 05:10:02,291 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 05:10:02,291 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 05:10:02,291 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:10:02,291 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29132.22 MB 2025-02-14 05:10:02,291 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29899.22 MB 2025-02-14 05:10:02,291 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 05:10:02,291 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37453.04 MB 2025-02-14 05:10:02,291 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37868.27 MB 2025-02-14 05:10:02,291 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 05:10:02,291 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30607.01 MB 2025-02-14 05:10:02,309 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 05:10:02,309 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 05:10:02,309 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:10:02,310 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:10:02,310 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30312.11 MB 2025-02-14 05:10:02,310 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30540.70 MB 2025-02-14 05:10:02,310 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.59 MB 2025-02-14 05:10:02,310 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37868.27 MB 2025-02-14 05:10:02,310 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37868.27 MB 2025-02-14 05:10:02,310 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:10:02,310 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30773.97 MB 2025-02-14 05:10:02,311 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 05:10:02,311 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 05:10:02,311 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.58 seconds 2025-02-14 05:10:02,311 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:10:02,311 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17766.28 MB 2025-02-14 05:10:02,311 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30740.59 MB 2025-02-14 05:10:02,311 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12974.31 MB 2025-02-14 05:10:02,311 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39271.27 MB 2025-02-14 05:10:02,311 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37868.27 MB 2025-02-14 05:10:02,311 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1402.99 MB 2025-02-14 05:10:02,311 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30773.97 MB 2025-02-14 05:10:02,577 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 05:10:02,577 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 05:10:02,577 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 05:10:02,577 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:10:02,577 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30740.59 MB 2025-02-14 05:10:02,577 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22752.39 MB 2025-02-14 05:10:02,577 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7988.20 MB 2025-02-14 05:10:02,577 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37868.27 MB 2025-02-14 05:10:02,577 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37868.27 MB 2025-02-14 05:10:02,577 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:10:02,577 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33237.51 MB 2025-02-14 05:10:02,594 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8114, cut from 8116 2025-02-14 05:10:02,595 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 05:10:02,601 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 05:10:02,601 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 05:10:02,601 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:10:02,601 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:10:02,601 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22752.39 MB 2025-02-14 05:10:02,601 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31141.53 MB 2025-02-14 05:10:02,601 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8389.15 MB 2025-02-14 05:10:02,601 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37868.27 MB 2025-02-14 05:10:02,601 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42039.51 MB 2025-02-14 05:10:02,601 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4171.24 MB 2025-02-14 05:10:02,601 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31141.53 MB 2025-02-14 05:10:02,763 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7906] 2025-02-14 05:10:02,765 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:10:02,765 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 05:10:02,766 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:10:02,766 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 05:10:02,771 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 05:10:02,772 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:10:02,772 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 05:10:02,772 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 05:11:11,407 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:11:11,407 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 05:11:11,412 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 05:11:11,416 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:11:11,416 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 806, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 05:11:11,417 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:11:11,417 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 806, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 05:11:23,763 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 05:11:23,764 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 05:11:23,764 - resource_logging.py:150 - __exit__ - DEBUG - Time: 12.34 seconds 2025-02-14 05:11:23,764 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:11:23,764 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18585.04 MB 2025-02-14 05:11:23,764 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21437.43 MB 2025-02-14 05:11:23,764 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2852.39 MB 2025-02-14 05:11:23,764 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50381.98 MB 2025-02-14 05:11:23,764 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28147.97 MB 2025-02-14 05:11:23,764 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22234.01 MB 2025-02-14 05:11:23,764 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30322.15 MB 2025-02-14 05:11:23,813 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 05:11:23,813 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 05:11:23,813 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-14 05:11:23,813 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:11:23,813 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21437.43 MB 2025-02-14 05:11:23,813 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19967.99 MB 2025-02-14 05:11:23,813 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1469.44 MB 2025-02-14 05:11:23,813 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28147.97 MB 2025-02-14 05:11:23,813 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35129.39 MB 2025-02-14 05:11:23,813 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6981.42 MB 2025-02-14 05:11:23,813 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31247.72 MB 2025-02-14 05:11:25,711 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 05:11:25,711 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 05:11:25,711 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.90 seconds 2025-02-14 05:11:25,711 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:11:25,711 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19967.99 MB 2025-02-14 05:11:25,711 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20498.84 MB 2025-02-14 05:11:25,711 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 05:11:25,711 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35129.39 MB 2025-02-14 05:11:25,712 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26709.33 MB 2025-02-14 05:11:25,712 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8420.07 MB 2025-02-14 05:11:25,712 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24478.17 MB 2025-02-14 05:11:25,725 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 05:11:25,725 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 05:11:25,725 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 05:11:25,725 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:11:25,725 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20498.84 MB 2025-02-14 05:11:25,725 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22388.37 MB 2025-02-14 05:11:25,725 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 05:11:25,725 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26709.33 MB 2025-02-14 05:11:25,725 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26709.33 MB 2025-02-14 05:11:25,725 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:11:25,725 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23805.80 MB 2025-02-14 05:11:25,929 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 05:11:25,929 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 05:11:25,929 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 05:11:25,929 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:11:25,929 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22388.37 MB 2025-02-14 05:11:25,929 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24630.23 MB 2025-02-14 05:11:25,929 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 05:11:25,929 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26709.33 MB 2025-02-14 05:11:25,929 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32843.50 MB 2025-02-14 05:11:25,929 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 05:11:25,929 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30174.51 MB 2025-02-14 05:11:25,930 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 05:11:25,930 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 05:11:25,930 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 05:11:25,930 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:11:25,930 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20498.84 MB 2025-02-14 05:11:25,930 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24630.23 MB 2025-02-14 05:11:25,930 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 05:11:25,930 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26709.33 MB 2025-02-14 05:11:25,930 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32843.50 MB 2025-02-14 05:11:25,930 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 05:11:25,930 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30174.51 MB 2025-02-14 05:11:26,087 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 05:11:26,087 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 05:11:26,087 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 05:11:26,087 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:11:26,087 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26163.77 MB 2025-02-14 05:11:26,087 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26930.77 MB 2025-02-14 05:11:26,087 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 05:11:26,087 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32843.50 MB 2025-02-14 05:11:26,087 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33258.73 MB 2025-02-14 05:11:26,087 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 05:11:26,087 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27638.56 MB 2025-02-14 05:11:26,105 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 05:11:26,105 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 05:11:26,105 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:11:26,105 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:11:26,105 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27343.66 MB 2025-02-14 05:11:26,105 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27571.53 MB 2025-02-14 05:11:26,105 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.87 MB 2025-02-14 05:11:26,105 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33258.73 MB 2025-02-14 05:11:26,105 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33258.73 MB 2025-02-14 05:11:26,105 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:11:26,105 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27783.78 MB 2025-02-14 05:11:26,106 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 05:11:26,106 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 05:11:26,106 - resource_logging.py:150 - __exit__ - DEBUG - Time: 14.69 seconds 2025-02-14 05:11:26,106 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:11:26,106 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15776.88 MB 2025-02-14 05:11:26,106 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27771.71 MB 2025-02-14 05:11:26,106 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11994.84 MB 2025-02-14 05:11:26,106 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50381.98 MB 2025-02-14 05:11:26,106 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33258.73 MB 2025-02-14 05:11:26,106 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17123.25 MB 2025-02-14 05:11:26,106 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27783.78 MB 2025-02-14 05:11:26,376 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 05:11:26,376 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 05:11:26,376 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 05:11:26,376 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:11:26,376 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27771.71 MB 2025-02-14 05:11:26,376 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20767.55 MB 2025-02-14 05:11:26,376 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7004.16 MB 2025-02-14 05:11:26,376 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33258.73 MB 2025-02-14 05:11:26,376 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33258.73 MB 2025-02-14 05:11:26,376 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:11:26,376 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30272.32 MB 2025-02-14 05:11:26,394 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8126, cut from 8128 2025-02-14 05:11:26,394 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-14 05:11:26,401 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 05:11:26,401 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 05:11:26,401 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:11:26,401 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:11:26,401 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20767.55 MB 2025-02-14 05:11:26,401 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29169.08 MB 2025-02-14 05:11:26,401 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8401.53 MB 2025-02-14 05:11:26,401 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33258.73 MB 2025-02-14 05:11:26,401 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41613.79 MB 2025-02-14 05:11:26,401 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8355.05 MB 2025-02-14 05:11:26,401 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29169.08 MB 2025-02-14 05:11:26,551 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7918] 2025-02-14 05:11:26,552 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:11:26,552 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 05:11:26,553 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:11:26,553 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 05:11:26,558 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 05:11:26,559 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:11:26,559 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 05:11:26,559 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-14 05:11:35,546 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:11:35,546 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 05:11:35,552 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 05:11:35,557 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:11:35,557 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1742, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 05:11:35,558 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:11:35,558 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1742, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 05:12:02,698 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 05:12:02,699 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 05:12:02,699 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.13 seconds 2025-02-14 05:12:02,699 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:12:02,699 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25107.24 MB 2025-02-14 05:12:02,699 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31272.87 MB 2025-02-14 05:12:02,699 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6165.63 MB 2025-02-14 05:12:02,699 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49968.84 MB 2025-02-14 05:12:02,699 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39785.07 MB 2025-02-14 05:12:02,699 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10183.77 MB 2025-02-14 05:12:02,699 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40241.73 MB 2025-02-14 05:12:02,794 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 05:12:02,794 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 05:12:02,794 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 05:12:02,794 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:12:02,794 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31272.87 MB 2025-02-14 05:12:02,794 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24833.97 MB 2025-02-14 05:12:02,794 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6438.90 MB 2025-02-14 05:12:02,794 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39785.07 MB 2025-02-14 05:12:02,794 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55253.66 MB 2025-02-14 05:12:02,794 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 15468.59 MB 2025-02-14 05:12:02,794 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47020.95 MB 2025-02-14 05:12:04,723 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 05:12:04,723 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 05:12:04,723 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 05:12:04,723 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:12:04,723 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24833.97 MB 2025-02-14 05:12:04,723 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25364.81 MB 2025-02-14 05:12:04,723 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 05:12:04,723 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55253.66 MB 2025-02-14 05:12:04,723 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30857.49 MB 2025-02-14 05:12:04,723 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24396.17 MB 2025-02-14 05:12:04,723 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29344.14 MB 2025-02-14 05:12:04,739 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 05:12:04,739 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 05:12:04,739 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 05:12:04,739 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:12:04,739 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25364.81 MB 2025-02-14 05:12:04,739 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27254.34 MB 2025-02-14 05:12:04,739 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 05:12:04,739 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30857.49 MB 2025-02-14 05:12:04,739 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31801.21 MB 2025-02-14 05:12:04,740 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 05:12:04,740 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28671.77 MB 2025-02-14 05:12:04,953 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 05:12:04,953 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 05:12:04,953 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 05:12:04,953 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:12:04,953 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27254.34 MB 2025-02-14 05:12:04,953 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29496.20 MB 2025-02-14 05:12:04,953 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 05:12:04,953 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31801.21 MB 2025-02-14 05:12:04,953 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37935.38 MB 2025-02-14 05:12:04,953 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 05:12:04,953 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35040.48 MB 2025-02-14 05:12:04,954 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 05:12:04,954 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 05:12:04,954 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 05:12:04,954 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:12:04,954 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25364.81 MB 2025-02-14 05:12:04,954 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29496.20 MB 2025-02-14 05:12:04,954 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 05:12:04,954 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30857.49 MB 2025-02-14 05:12:04,954 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37935.38 MB 2025-02-14 05:12:04,954 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7077.89 MB 2025-02-14 05:12:04,954 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35040.48 MB 2025-02-14 05:12:05,119 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 05:12:05,119 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 05:12:05,119 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 05:12:05,119 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:12:05,119 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31029.74 MB 2025-02-14 05:12:05,119 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31796.74 MB 2025-02-14 05:12:05,119 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 05:12:05,119 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37935.38 MB 2025-02-14 05:12:05,119 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38350.62 MB 2025-02-14 05:12:05,119 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 05:12:05,119 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32504.53 MB 2025-02-14 05:12:05,138 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 05:12:05,138 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 05:12:05,138 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:12:05,138 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:12:05,138 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32209.63 MB 2025-02-14 05:12:05,138 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32436.99 MB 2025-02-14 05:12:05,138 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.36 MB 2025-02-14 05:12:05,138 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38350.62 MB 2025-02-14 05:12:05,138 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38350.62 MB 2025-02-14 05:12:05,139 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:12:05,139 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32677.07 MB 2025-02-14 05:12:05,140 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 05:12:05,140 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 05:12:05,140 - resource_logging.py:150 - __exit__ - DEBUG - Time: 29.58 seconds 2025-02-14 05:12:05,140 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:12:05,140 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19037.97 MB 2025-02-14 05:12:05,140 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32637.21 MB 2025-02-14 05:12:05,140 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13599.23 MB 2025-02-14 05:12:05,140 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49968.84 MB 2025-02-14 05:12:05,140 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38350.62 MB 2025-02-14 05:12:05,140 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11618.22 MB 2025-02-14 05:12:05,140 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32677.07 MB 2025-02-14 05:12:05,410 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 05:12:05,410 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 05:12:05,410 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 05:12:05,410 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:12:05,410 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32637.21 MB 2025-02-14 05:12:05,410 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24029.03 MB 2025-02-14 05:12:05,410 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8608.18 MB 2025-02-14 05:12:05,410 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38350.62 MB 2025-02-14 05:12:05,410 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38350.62 MB 2025-02-14 05:12:05,410 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:12:05,410 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35138.12 MB 2025-02-14 05:12:05,428 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8127, cut from 8129 2025-02-14 05:12:05,428 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-14 05:12:05,434 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 05:12:05,434 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 05:12:05,434 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:12:05,434 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:12:05,434 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24029.03 MB 2025-02-14 05:12:05,434 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32432.59 MB 2025-02-14 05:12:05,434 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8403.56 MB 2025-02-14 05:12:05,434 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38350.62 MB 2025-02-14 05:12:05,434 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42528.15 MB 2025-02-14 05:12:05,434 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4177.53 MB 2025-02-14 05:12:05,434 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32432.59 MB 2025-02-14 05:12:05,596 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7919] 2025-02-14 05:12:05,597 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:12:05,597 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 05:12:05,598 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:12:05,598 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 05:12:05,603 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 05:12:05,604 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:12:05,604 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 05:12:05,604 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-14 05:13:00,069 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:13:00,069 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 05:13:00,074 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 05:13:00,078 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:13:00,079 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 184, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 05:13:00,079 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:13:00,080 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 184, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 05:13:02,923 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 05:13:02,924 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 05:13:02,924 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.84 seconds 2025-02-14 05:13:02,924 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:13:02,924 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14250.85 MB 2025-02-14 05:13:02,924 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14902.01 MB 2025-02-14 05:13:02,924 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 651.17 MB 2025-02-14 05:13:02,924 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50883.20 MB 2025-02-14 05:13:02,924 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17855.15 MB 2025-02-14 05:13:02,924 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33028.05 MB 2025-02-14 05:13:02,924 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23723.03 MB 2025-02-14 05:13:02,933 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 05:13:02,933 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 05:13:02,933 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 05:13:02,933 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:13:02,933 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14902.01 MB 2025-02-14 05:13:02,933 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14009.54 MB 2025-02-14 05:13:02,933 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -892.47 MB 2025-02-14 05:13:02,933 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17855.15 MB 2025-02-14 05:13:02,933 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17855.15 MB 2025-02-14 05:13:02,933 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:13:02,933 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15098.94 MB 2025-02-14 05:13:03,004 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 05:13:03,004 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 05:13:03,004 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 05:13:03,004 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:13:03,004 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14009.54 MB 2025-02-14 05:13:03,004 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14025.47 MB 2025-02-14 05:13:03,005 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 15.93 MB 2025-02-14 05:13:03,005 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17855.15 MB 2025-02-14 05:13:03,005 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17855.15 MB 2025-02-14 05:13:03,005 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:13:03,005 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14775.43 MB 2025-02-14 05:13:03,009 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 05:13:03,009 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 05:13:03,009 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 05:13:03,009 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:13:03,009 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14025.40 MB 2025-02-14 05:13:03,009 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14082.86 MB 2025-02-14 05:13:03,009 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 57.46 MB 2025-02-14 05:13:03,009 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17855.15 MB 2025-02-14 05:13:03,009 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17855.15 MB 2025-02-14 05:13:03,009 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:13:03,009 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14126.05 MB 2025-02-14 05:13:03,021 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 05:13:03,021 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 05:13:03,021 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 05:13:03,021 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:13:03,021 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14082.86 MB 2025-02-14 05:13:03,021 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14150.18 MB 2025-02-14 05:13:03,021 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 67.32 MB 2025-02-14 05:13:03,021 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17855.15 MB 2025-02-14 05:13:03,021 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17855.15 MB 2025-02-14 05:13:03,021 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:13:03,021 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14318.14 MB 2025-02-14 05:13:03,022 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 05:13:03,022 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 05:13:03,022 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:13:03,022 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:13:03,022 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14025.40 MB 2025-02-14 05:13:03,022 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14150.18 MB 2025-02-14 05:13:03,022 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 124.78 MB 2025-02-14 05:13:03,022 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17855.15 MB 2025-02-14 05:13:03,022 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17855.15 MB 2025-02-14 05:13:03,022 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:13:03,022 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14318.14 MB 2025-02-14 05:13:03,030 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 05:13:03,031 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 05:13:03,031 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 05:13:03,031 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:13:03,031 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14196.84 MB 2025-02-14 05:13:03,031 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14220.89 MB 2025-02-14 05:13:03,031 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 24.05 MB 2025-02-14 05:13:03,031 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17855.15 MB 2025-02-14 05:13:03,031 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17861.44 MB 2025-02-14 05:13:03,031 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6.29 MB 2025-02-14 05:13:03,031 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14251.23 MB 2025-02-14 05:13:03,033 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 05:13:03,033 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 05:13:03,033 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 05:13:03,033 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:13:03,033 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14233.29 MB 2025-02-14 05:13:03,033 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14250.51 MB 2025-02-14 05:13:03,033 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 17.22 MB 2025-02-14 05:13:03,033 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17861.44 MB 2025-02-14 05:13:03,033 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17861.44 MB 2025-02-14 05:13:03,033 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:13:03,033 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14250.51 MB 2025-02-14 05:13:03,034 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 05:13:03,034 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 05:13:03,034 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.95 seconds 2025-02-14 05:13:03,034 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:13:03,034 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13609.78 MB 2025-02-14 05:13:03,034 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14281.12 MB 2025-02-14 05:13:03,034 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 671.34 MB 2025-02-14 05:13:03,035 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50883.20 MB 2025-02-14 05:13:03,035 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17861.44 MB 2025-02-14 05:13:03,035 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33021.76 MB 2025-02-14 05:13:03,035 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14281.12 MB 2025-02-14 05:13:03,094 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 05:13:03,094 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 05:13:03,094 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 05:13:03,094 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:13:03,094 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14281.12 MB 2025-02-14 05:13:03,094 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14740.08 MB 2025-02-14 05:13:03,094 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 458.96 MB 2025-02-14 05:13:03,094 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17861.44 MB 2025-02-14 05:13:03,094 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17863.54 MB 2025-02-14 05:13:03,094 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-14 05:13:03,094 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14785.97 MB 2025-02-14 05:13:03,098 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 1231, cut from 1233 2025-02-14 05:13:03,099 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-14 05:13:03,100 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 05:13:03,101 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 05:13:03,101 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 05:13:03,101 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:13:03,101 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14144.28 MB 2025-02-14 05:13:03,101 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15428.84 MB 2025-02-14 05:13:03,101 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1284.57 MB 2025-02-14 05:13:03,101 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17863.54 MB 2025-02-14 05:13:03,101 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17863.54 MB 2025-02-14 05:13:03,101 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:13:03,101 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15428.84 MB 2025-02-14 05:13:03,126 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 1023] 2025-02-14 05:13:03,127 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:13:03,127 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 05:13:03,128 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:13:03,128 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 05:13:03,133 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 05:13:03,134 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:13:03,134 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 05:13:03,134 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-14 05:13:48,402 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:13:48,402 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 05:13:48,407 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 05:13:48,411 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:13:48,411 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1217, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 05:13:48,412 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:13:48,412 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1217, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 05:14:07,128 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 05:14:07,128 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 05:14:07,128 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.71 seconds 2025-02-14 05:14:07,128 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:14:07,128 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21449.61 MB 2025-02-14 05:14:07,128 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25757.16 MB 2025-02-14 05:14:07,128 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4307.55 MB 2025-02-14 05:14:07,128 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26348.62 MB 2025-02-14 05:14:07,128 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29704.06 MB 2025-02-14 05:14:07,128 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3355.44 MB 2025-02-14 05:14:07,128 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34772.16 MB 2025-02-14 05:14:07,209 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 05:14:07,209 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 05:14:07,209 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 05:14:07,209 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:14:07,209 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25757.16 MB 2025-02-14 05:14:07,209 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22104.98 MB 2025-02-14 05:14:07,209 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3652.18 MB 2025-02-14 05:14:07,209 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29704.06 MB 2025-02-14 05:14:07,209 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46045.07 MB 2025-02-14 05:14:07,209 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 16341.01 MB 2025-02-14 05:14:07,209 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38615.64 MB 2025-02-14 05:14:09,149 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 05:14:09,149 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 05:14:09,149 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-14 05:14:09,149 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:14:09,149 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22104.98 MB 2025-02-14 05:14:09,149 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22635.82 MB 2025-02-14 05:14:09,149 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 05:14:09,149 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46045.07 MB 2025-02-14 05:14:09,149 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24658.31 MB 2025-02-14 05:14:09,149 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21386.76 MB 2025-02-14 05:14:09,149 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26616.20 MB 2025-02-14 05:14:09,163 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 05:14:09,163 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 05:14:09,163 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 05:14:09,163 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:14:09,163 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22635.82 MB 2025-02-14 05:14:09,163 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24525.35 MB 2025-02-14 05:14:09,163 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.52 MB 2025-02-14 05:14:09,163 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24658.31 MB 2025-02-14 05:14:09,163 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27961.33 MB 2025-02-14 05:14:09,163 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 05:14:09,163 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25942.77 MB 2025-02-14 05:14:09,374 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 05:14:09,374 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 05:14:09,374 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 05:14:09,374 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:14:09,374 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24525.35 MB 2025-02-14 05:14:09,374 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26767.20 MB 2025-02-14 05:14:09,374 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 05:14:09,374 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27961.33 MB 2025-02-14 05:14:09,374 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34567.36 MB 2025-02-14 05:14:09,374 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 05:14:09,374 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32311.48 MB 2025-02-14 05:14:09,375 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 05:14:09,375 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 05:14:09,375 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 05:14:09,375 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:14:09,375 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22635.82 MB 2025-02-14 05:14:09,375 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26767.20 MB 2025-02-14 05:14:09,375 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.38 MB 2025-02-14 05:14:09,375 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24658.31 MB 2025-02-14 05:14:09,375 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34567.36 MB 2025-02-14 05:14:09,375 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9909.04 MB 2025-02-14 05:14:09,375 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32311.48 MB 2025-02-14 05:14:09,546 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 05:14:09,546 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 05:14:09,546 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 05:14:09,546 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:14:09,546 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28300.74 MB 2025-02-14 05:14:09,546 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29067.75 MB 2025-02-14 05:14:09,546 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 05:14:09,546 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34567.36 MB 2025-02-14 05:14:09,546 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34980.50 MB 2025-02-14 05:14:09,546 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 05:14:09,546 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29775.53 MB 2025-02-14 05:14:09,565 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 05:14:09,565 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 05:14:09,565 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:14:09,565 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:14:09,565 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29480.63 MB 2025-02-14 05:14:09,565 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29708.94 MB 2025-02-14 05:14:09,565 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.31 MB 2025-02-14 05:14:09,565 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34980.50 MB 2025-02-14 05:14:09,565 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34980.50 MB 2025-02-14 05:14:09,565 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:14:09,565 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29945.26 MB 2025-02-14 05:14:09,566 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 05:14:09,566 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 05:14:09,566 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.15 seconds 2025-02-14 05:14:09,566 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:14:09,566 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17209.16 MB 2025-02-14 05:14:09,566 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29909.45 MB 2025-02-14 05:14:09,566 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12700.29 MB 2025-02-14 05:14:09,566 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22106.08 MB 2025-02-14 05:14:09,566 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34980.50 MB 2025-02-14 05:14:09,566 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 12874.42 MB 2025-02-14 05:14:09,566 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29945.26 MB 2025-02-14 05:14:09,835 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 05:14:09,835 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 05:14:09,835 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 05:14:09,835 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:14:09,835 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29909.45 MB 2025-02-14 05:14:09,835 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22204.78 MB 2025-02-14 05:14:09,835 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7704.68 MB 2025-02-14 05:14:09,835 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34980.50 MB 2025-02-14 05:14:09,835 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34980.50 MB 2025-02-14 05:14:09,835 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:14:09,835 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32414.05 MB 2025-02-14 05:14:09,853 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8139, cut from 8141 2025-02-14 05:14:09,853 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2 ('] 2025-02-14 05:14:09,859 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 05:14:09,859 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 05:14:09,859 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:14:09,860 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:14:09,860 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22204.78 MB 2025-02-14 05:14:09,860 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30619.73 MB 2025-02-14 05:14:09,860 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8414.95 MB 2025-02-14 05:14:09,860 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34980.50 MB 2025-02-14 05:14:09,860 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45441.09 MB 2025-02-14 05:14:09,860 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10460.59 MB 2025-02-14 05:14:09,860 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30619.73 MB 2025-02-14 05:14:10,028 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7931] 2025-02-14 05:14:10,030 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:14:10,030 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 05:14:10,031 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:14:10,031 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 05:14:10,035 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 05:14:10,037 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:14:10,037 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 05:14:10,037 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2 ('] 2025-02-14 05:14:58,696 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:14:58,696 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 05:14:58,701 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 05:14:58,704 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:14:58,704 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1005, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 05:14:58,705 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:14:58,705 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1005, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 05:15:14,153 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 05:15:14,153 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 05:15:14,153 - resource_logging.py:150 - __exit__ - DEBUG - Time: 15.44 seconds 2025-02-14 05:15:14,153 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:15:14,153 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19971.71 MB 2025-02-14 05:15:14,153 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23528.48 MB 2025-02-14 05:15:14,153 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3556.77 MB 2025-02-14 05:15:14,153 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53808.73 MB 2025-02-14 05:15:14,153 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28892.46 MB 2025-02-14 05:15:14,153 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24916.26 MB 2025-02-14 05:15:14,153 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32387.48 MB 2025-02-14 05:15:14,216 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 05:15:14,216 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 05:15:14,217 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 05:15:14,217 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:15:14,217 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23528.48 MB 2025-02-14 05:15:14,217 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21002.53 MB 2025-02-14 05:15:14,217 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2525.94 MB 2025-02-14 05:15:14,217 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28892.46 MB 2025-02-14 05:15:14,217 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40783.31 MB 2025-02-14 05:15:14,217 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 11890.85 MB 2025-02-14 05:15:14,217 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34858.28 MB 2025-02-14 05:15:16,128 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 05:15:16,128 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 05:15:16,128 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 05:15:16,128 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:15:16,128 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21002.53 MB 2025-02-14 05:15:16,128 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21533.37 MB 2025-02-14 05:15:16,128 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 05:15:16,128 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40783.31 MB 2025-02-14 05:15:16,128 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24662.51 MB 2025-02-14 05:15:16,128 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16120.81 MB 2025-02-14 05:15:16,128 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25512.71 MB 2025-02-14 05:15:16,142 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 05:15:16,142 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 05:15:16,142 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 05:15:16,142 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:15:16,142 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21533.37 MB 2025-02-14 05:15:16,142 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23422.91 MB 2025-02-14 05:15:16,142 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 05:15:16,142 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24662.51 MB 2025-02-14 05:15:16,142 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27965.52 MB 2025-02-14 05:15:16,142 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 05:15:16,142 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24840.34 MB 2025-02-14 05:15:16,354 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 05:15:16,354 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 05:15:16,354 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 05:15:16,354 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:15:16,354 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23422.91 MB 2025-02-14 05:15:16,354 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25664.76 MB 2025-02-14 05:15:16,354 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 05:15:16,354 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27965.52 MB 2025-02-14 05:15:16,354 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34099.69 MB 2025-02-14 05:15:16,354 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 05:15:16,354 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31209.04 MB 2025-02-14 05:15:16,355 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 05:15:16,355 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 05:15:16,355 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 05:15:16,355 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:15:16,355 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21533.37 MB 2025-02-14 05:15:16,355 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25664.76 MB 2025-02-14 05:15:16,355 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 05:15:16,355 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24662.51 MB 2025-02-14 05:15:16,355 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34099.69 MB 2025-02-14 05:15:16,355 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9437.18 MB 2025-02-14 05:15:16,355 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31209.04 MB 2025-02-14 05:15:16,520 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 05:15:16,520 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 05:15:16,520 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 05:15:16,520 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:15:16,520 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27198.31 MB 2025-02-14 05:15:16,520 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27965.31 MB 2025-02-14 05:15:16,520 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 05:15:16,520 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34099.69 MB 2025-02-14 05:15:16,520 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34512.83 MB 2025-02-14 05:15:16,520 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 05:15:16,520 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28673.10 MB 2025-02-14 05:15:16,538 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 05:15:16,538 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 05:15:16,538 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:15:16,538 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:15:16,538 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28378.20 MB 2025-02-14 05:15:16,538 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28606.21 MB 2025-02-14 05:15:16,538 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.02 MB 2025-02-14 05:15:16,538 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34512.83 MB 2025-02-14 05:15:16,538 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34512.83 MB 2025-02-14 05:15:16,538 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:15:16,538 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28842.44 MB 2025-02-14 05:15:16,540 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 05:15:16,540 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 05:15:16,540 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.83 seconds 2025-02-14 05:15:16,540 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:15:16,540 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16470.21 MB 2025-02-14 05:15:16,540 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28807.24 MB 2025-02-14 05:15:16,540 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12337.03 MB 2025-02-14 05:15:16,540 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53808.73 MB 2025-02-14 05:15:16,540 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34512.83 MB 2025-02-14 05:15:16,540 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19295.90 MB 2025-02-14 05:15:16,540 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28842.44 MB 2025-02-14 05:15:16,807 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 05:15:16,807 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 05:15:16,807 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 05:15:16,807 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:15:16,807 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28807.24 MB 2025-02-14 05:15:16,807 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21473.83 MB 2025-02-14 05:15:16,808 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7333.40 MB 2025-02-14 05:15:16,808 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34512.83 MB 2025-02-14 05:15:16,808 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34512.83 MB 2025-02-14 05:15:16,808 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:15:16,808 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31318.29 MB 2025-02-14 05:15:16,826 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8160, cut from 8162 2025-02-14 05:15:16,826 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2 ('] 2025-02-14 05:15:16,832 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 05:15:16,832 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 05:15:16,832 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:15:16,832 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:15:16,832 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21473.83 MB 2025-02-14 05:15:16,832 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29911.31 MB 2025-02-14 05:15:16,832 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8437.47 MB 2025-02-14 05:15:16,832 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34512.83 MB 2025-02-14 05:15:16,832 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44998.59 MB 2025-02-14 05:15:16,832 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10485.76 MB 2025-02-14 05:15:16,832 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29911.31 MB 2025-02-14 05:15:16,996 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7952] 2025-02-14 05:15:16,998 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:15:16,998 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 05:15:16,999 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:15:16,999 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 05:15:17,004 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 05:15:17,005 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:15:17,005 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 05:15:17,005 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2 ('] 2025-02-14 05:16:10,477 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:16:10,477 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 05:16:10,483 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 05:16:10,486 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:16:10,486 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1098, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 05:16:10,487 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:16:10,487 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1098, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 05:16:27,397 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 05:16:27,397 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 05:16:27,397 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.90 seconds 2025-02-14 05:16:27,397 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:16:27,397 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20619.75 MB 2025-02-14 05:16:27,398 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24505.77 MB 2025-02-14 05:16:27,398 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3886.02 MB 2025-02-14 05:16:27,398 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53387.20 MB 2025-02-14 05:16:27,398 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29232.20 MB 2025-02-14 05:16:27,398 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24155.00 MB 2025-02-14 05:16:27,398 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33489.31 MB 2025-02-14 05:16:27,468 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 05:16:27,468 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 05:16:27,468 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 05:16:27,468 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:16:27,468 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24505.77 MB 2025-02-14 05:16:27,468 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21486.01 MB 2025-02-14 05:16:27,468 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3019.76 MB 2025-02-14 05:16:27,468 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29232.20 MB 2025-02-14 05:16:27,468 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43037.75 MB 2025-02-14 05:16:27,468 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 13805.55 MB 2025-02-14 05:16:27,468 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36421.70 MB 2025-02-14 05:16:29,397 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 05:16:29,397 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 05:16:29,397 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 05:16:29,397 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:16:29,397 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21486.01 MB 2025-02-14 05:16:29,397 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22016.85 MB 2025-02-14 05:16:29,397 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 05:16:29,397 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43037.75 MB 2025-02-14 05:16:29,397 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24672.99 MB 2025-02-14 05:16:29,397 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18364.76 MB 2025-02-14 05:16:29,397 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25997.22 MB 2025-02-14 05:16:29,410 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 05:16:29,410 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 05:16:29,411 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 05:16:29,411 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:16:29,411 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22016.85 MB 2025-02-14 05:16:29,411 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23906.39 MB 2025-02-14 05:16:29,411 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 05:16:29,411 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24672.99 MB 2025-02-14 05:16:29,411 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27976.01 MB 2025-02-14 05:16:29,411 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 05:16:29,411 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25323.81 MB 2025-02-14 05:16:29,618 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 05:16:29,618 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 05:16:29,618 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 05:16:29,618 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:16:29,618 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23906.39 MB 2025-02-14 05:16:29,618 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26148.24 MB 2025-02-14 05:16:29,618 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 05:16:29,618 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27976.01 MB 2025-02-14 05:16:29,618 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34110.18 MB 2025-02-14 05:16:29,618 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 05:16:29,618 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31692.52 MB 2025-02-14 05:16:29,619 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 05:16:29,619 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 05:16:29,619 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 05:16:29,619 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:16:29,619 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22016.85 MB 2025-02-14 05:16:29,619 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26148.24 MB 2025-02-14 05:16:29,619 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 05:16:29,619 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24672.99 MB 2025-02-14 05:16:29,619 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34110.18 MB 2025-02-14 05:16:29,619 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9437.18 MB 2025-02-14 05:16:29,619 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31692.52 MB 2025-02-14 05:16:29,783 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 05:16:29,783 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 05:16:29,783 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 05:16:29,783 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:16:29,783 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27681.78 MB 2025-02-14 05:16:29,783 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28448.79 MB 2025-02-14 05:16:29,783 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 05:16:29,783 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34110.18 MB 2025-02-14 05:16:29,783 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34521.22 MB 2025-02-14 05:16:29,783 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 411.04 MB 2025-02-14 05:16:29,783 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29156.57 MB 2025-02-14 05:16:29,802 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 05:16:29,802 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 05:16:29,802 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:16:29,802 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:16:29,802 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28861.68 MB 2025-02-14 05:16:29,802 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29089.37 MB 2025-02-14 05:16:29,802 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.70 MB 2025-02-14 05:16:29,802 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34521.22 MB 2025-02-14 05:16:29,802 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34521.22 MB 2025-02-14 05:16:29,802 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:16:29,802 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29326.08 MB 2025-02-14 05:16:29,803 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 05:16:29,803 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 05:16:29,803 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.31 seconds 2025-02-14 05:16:29,803 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:16:29,803 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16794.23 MB 2025-02-14 05:16:29,803 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29289.85 MB 2025-02-14 05:16:29,803 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12495.63 MB 2025-02-14 05:16:29,803 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53387.20 MB 2025-02-14 05:16:29,803 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34521.22 MB 2025-02-14 05:16:29,803 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18865.98 MB 2025-02-14 05:16:29,803 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29326.08 MB 2025-02-14 05:16:30,070 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 05:16:30,071 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 05:16:30,071 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 05:16:30,071 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:16:30,071 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29289.85 MB 2025-02-14 05:16:30,071 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21789.47 MB 2025-02-14 05:16:30,071 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7500.38 MB 2025-02-14 05:16:30,071 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34521.22 MB 2025-02-14 05:16:30,071 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34521.22 MB 2025-02-14 05:16:30,071 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:16:30,071 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31794.15 MB 2025-02-14 05:16:30,089 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8138, cut from 8140 2025-02-14 05:16:30,089 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 05:16:30,095 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 05:16:30,095 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 05:16:30,095 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:16:30,096 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:16:30,096 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21789.47 MB 2025-02-14 05:16:30,096 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30203.45 MB 2025-02-14 05:16:30,096 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8413.98 MB 2025-02-14 05:16:30,096 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34521.22 MB 2025-02-14 05:16:30,096 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44979.72 MB 2025-02-14 05:16:30,096 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10458.50 MB 2025-02-14 05:16:30,096 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30203.45 MB 2025-02-14 05:16:30,258 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7930] 2025-02-14 05:16:30,259 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:16:30,260 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 05:16:30,260 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:16:30,261 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 05:16:30,265 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 05:16:30,266 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:16:30,266 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 05:16:30,266 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 05:16:47,502 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:16:47,502 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 05:16:47,507 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 05:16:47,511 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:16:47,511 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1131, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 05:16:47,512 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:16:47,512 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1131, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 05:17:05,060 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 05:17:05,060 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 05:17:05,060 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.54 seconds 2025-02-14 05:17:05,060 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:17:05,060 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20849.70 MB 2025-02-14 05:17:05,060 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24853.16 MB 2025-02-14 05:17:05,060 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4003.46 MB 2025-02-14 05:17:05,060 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57526.98 MB 2025-02-14 05:17:05,060 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31379.69 MB 2025-02-14 05:17:05,060 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26147.29 MB 2025-02-14 05:17:05,060 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33718.45 MB 2025-02-14 05:17:05,131 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 05:17:05,131 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 05:17:05,131 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 05:17:05,131 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:17:05,131 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24853.16 MB 2025-02-14 05:17:05,131 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21657.57 MB 2025-02-14 05:17:05,131 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3195.59 MB 2025-02-14 05:17:05,131 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31379.69 MB 2025-02-14 05:17:05,131 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42362.47 MB 2025-02-14 05:17:05,131 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10982.79 MB 2025-02-14 05:17:05,131 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36991.68 MB 2025-02-14 05:17:07,057 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 05:17:07,057 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 05:17:07,057 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 05:17:07,057 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:17:07,057 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21657.57 MB 2025-02-14 05:17:07,057 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22188.41 MB 2025-02-14 05:17:07,057 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 05:17:07,057 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42362.47 MB 2025-02-14 05:17:07,057 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26698.84 MB 2025-02-14 05:17:07,057 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15663.63 MB 2025-02-14 05:17:07,057 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26168.78 MB 2025-02-14 05:17:07,071 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 05:17:07,071 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 05:17:07,071 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 05:17:07,071 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:17:07,071 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22188.41 MB 2025-02-14 05:17:07,071 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24077.94 MB 2025-02-14 05:17:07,071 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 05:17:07,071 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26698.84 MB 2025-02-14 05:17:07,071 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28586.28 MB 2025-02-14 05:17:07,071 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 05:17:07,071 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25495.37 MB 2025-02-14 05:17:07,281 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 05:17:07,281 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 05:17:07,281 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 05:17:07,281 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:17:07,281 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24077.94 MB 2025-02-14 05:17:07,281 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26319.80 MB 2025-02-14 05:17:07,281 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 05:17:07,281 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28586.28 MB 2025-02-14 05:17:07,281 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34248.59 MB 2025-02-14 05:17:07,281 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 05:17:07,281 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31864.08 MB 2025-02-14 05:17:07,282 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 05:17:07,282 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 05:17:07,282 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 05:17:07,282 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:17:07,282 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22188.41 MB 2025-02-14 05:17:07,282 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26319.80 MB 2025-02-14 05:17:07,282 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 05:17:07,282 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26698.84 MB 2025-02-14 05:17:07,282 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34248.59 MB 2025-02-14 05:17:07,282 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-14 05:17:07,282 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31864.08 MB 2025-02-14 05:17:07,446 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 05:17:07,446 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 05:17:07,446 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 05:17:07,446 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:17:07,446 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27853.34 MB 2025-02-14 05:17:07,446 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28620.34 MB 2025-02-14 05:17:07,446 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 05:17:07,446 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34248.59 MB 2025-02-14 05:17:07,447 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34661.73 MB 2025-02-14 05:17:07,447 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 05:17:07,447 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29328.13 MB 2025-02-14 05:17:07,465 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 05:17:07,465 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 05:17:07,465 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:17:07,465 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:17:07,465 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29033.23 MB 2025-02-14 05:17:07,465 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29262.36 MB 2025-02-14 05:17:07,465 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.13 MB 2025-02-14 05:17:07,465 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34661.73 MB 2025-02-14 05:17:07,465 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34661.73 MB 2025-02-14 05:17:07,465 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:17:07,465 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29497.32 MB 2025-02-14 05:17:07,466 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 05:17:07,466 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 05:17:07,466 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.95 seconds 2025-02-14 05:17:07,466 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:17:07,466 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16909.20 MB 2025-02-14 05:17:07,466 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29462.95 MB 2025-02-14 05:17:07,466 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12553.74 MB 2025-02-14 05:17:07,466 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57526.98 MB 2025-02-14 05:17:07,466 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34661.73 MB 2025-02-14 05:17:07,466 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22865.25 MB 2025-02-14 05:17:07,466 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29497.32 MB 2025-02-14 05:17:07,735 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 05:17:07,735 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 05:17:07,735 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 05:17:07,735 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:17:07,735 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29462.95 MB 2025-02-14 05:17:07,735 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21905.97 MB 2025-02-14 05:17:07,735 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7556.97 MB 2025-02-14 05:17:07,735 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34661.73 MB 2025-02-14 05:17:07,735 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34661.73 MB 2025-02-14 05:17:07,735 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:17:07,735 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31968.47 MB 2025-02-14 05:17:07,753 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8142, cut from 8144 2025-02-14 05:17:07,753 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 05:17:07,759 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 05:17:07,759 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 05:17:07,759 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:17:07,759 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:17:07,759 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21905.97 MB 2025-02-14 05:17:07,759 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30324.13 MB 2025-02-14 05:17:07,759 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8418.15 MB 2025-02-14 05:17:07,759 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34661.73 MB 2025-02-14 05:17:07,759 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45124.42 MB 2025-02-14 05:17:07,759 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10462.69 MB 2025-02-14 05:17:07,759 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30324.13 MB 2025-02-14 05:17:07,921 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7934] 2025-02-14 05:17:07,922 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:17:07,922 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 05:17:07,923 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:17:07,923 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 05:17:07,928 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 05:17:07,929 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:17:07,929 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 05:17:07,929 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 05:17:25,257 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:17:25,257 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 05:17:25,262 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 05:17:25,266 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:17:25,266 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 310, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 05:17:25,267 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:17:25,267 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 310, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 05:17:30,122 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 05:17:30,122 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 05:17:30,122 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.85 seconds 2025-02-14 05:17:30,122 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:17:30,122 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15128.84 MB 2025-02-14 05:17:30,122 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16225.91 MB 2025-02-14 05:17:30,122 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1097.07 MB 2025-02-14 05:17:30,122 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57677.97 MB 2025-02-14 05:17:30,122 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19897.78 MB 2025-02-14 05:17:30,122 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -37780.19 MB 2025-02-14 05:17:30,122 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25054.00 MB 2025-02-14 05:17:30,142 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 05:17:30,142 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 05:17:30,142 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:17:30,142 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:17:30,142 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16225.91 MB 2025-02-14 05:17:30,142 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16702.55 MB 2025-02-14 05:17:30,142 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 476.64 MB 2025-02-14 05:17:30,142 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19897.78 MB 2025-02-14 05:17:30,142 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23089.64 MB 2025-02-14 05:17:30,142 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3191.87 MB 2025-02-14 05:17:30,142 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20491.20 MB 2025-02-14 05:17:31,591 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 05:17:31,591 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 05:17:31,591 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.45 seconds 2025-02-14 05:17:31,591 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:17:31,591 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16702.55 MB 2025-02-14 05:17:31,591 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17103.34 MB 2025-02-14 05:17:31,591 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 400.79 MB 2025-02-14 05:17:31,591 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23089.64 MB 2025-02-14 05:17:31,591 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20921.19 MB 2025-02-14 05:17:31,591 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2168.46 MB 2025-02-14 05:17:31,591 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21043.90 MB 2025-02-14 05:17:31,602 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 05:17:31,602 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 05:17:31,602 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 05:17:31,602 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:17:31,602 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17103.34 MB 2025-02-14 05:17:31,602 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18530.97 MB 2025-02-14 05:17:31,602 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1427.64 MB 2025-02-14 05:17:31,602 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20921.19 MB 2025-02-14 05:17:31,602 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21634.22 MB 2025-02-14 05:17:31,602 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 713.03 MB 2025-02-14 05:17:31,602 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19601.13 MB 2025-02-14 05:17:31,760 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 05:17:31,760 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 05:17:31,760 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 05:17:31,760 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:17:31,760 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18530.97 MB 2025-02-14 05:17:31,760 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20224.12 MB 2025-02-14 05:17:31,760 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1693.14 MB 2025-02-14 05:17:31,760 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21634.22 MB 2025-02-14 05:17:31,760 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25912.41 MB 2025-02-14 05:17:31,760 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4278.19 MB 2025-02-14 05:17:31,760 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24412.13 MB 2025-02-14 05:17:31,761 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 05:17:31,761 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 05:17:31,761 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 05:17:31,761 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:17:31,761 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17103.34 MB 2025-02-14 05:17:31,761 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20224.12 MB 2025-02-14 05:17:31,761 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3120.78 MB 2025-02-14 05:17:31,761 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20921.19 MB 2025-02-14 05:17:31,761 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25912.41 MB 2025-02-14 05:17:31,761 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4991.22 MB 2025-02-14 05:17:31,761 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24412.13 MB 2025-02-14 05:17:31,925 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 05:17:31,925 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 05:17:31,925 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 05:17:31,925 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:17:31,925 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21381.94 MB 2025-02-14 05:17:31,925 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21961.03 MB 2025-02-14 05:17:31,925 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 579.09 MB 2025-02-14 05:17:31,925 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25912.41 MB 2025-02-14 05:17:31,925 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26222.79 MB 2025-02-14 05:17:31,925 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 310.38 MB 2025-02-14 05:17:31,925 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22495.41 MB 2025-02-14 05:17:31,940 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 05:17:31,940 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 05:17:31,940 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 05:17:31,940 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:17:31,940 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22272.76 MB 2025-02-14 05:17:31,940 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22494.21 MB 2025-02-14 05:17:31,940 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 221.45 MB 2025-02-14 05:17:31,940 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26222.79 MB 2025-02-14 05:17:31,940 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26222.79 MB 2025-02-14 05:17:31,940 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:17:31,940 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22600.51 MB 2025-02-14 05:17:31,941 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 05:17:31,941 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 05:17:31,941 - resource_logging.py:150 - __exit__ - DEBUG - Time: 6.67 seconds 2025-02-14 05:17:31,941 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:17:31,941 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14048.77 MB 2025-02-14 05:17:31,941 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22694.86 MB 2025-02-14 05:17:31,941 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8646.09 MB 2025-02-14 05:17:31,941 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57677.97 MB 2025-02-14 05:17:31,941 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26222.79 MB 2025-02-14 05:17:31,941 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31455.18 MB 2025-02-14 05:17:31,941 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22694.86 MB 2025-02-14 05:17:32,212 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 05:17:32,212 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 05:17:32,212 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 05:17:32,212 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:17:32,212 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22694.86 MB 2025-02-14 05:17:32,212 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25702.63 MB 2025-02-14 05:17:32,212 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3007.77 MB 2025-02-14 05:17:32,212 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26222.79 MB 2025-02-14 05:17:32,212 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26893.88 MB 2025-02-14 05:17:32,212 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 671.09 MB 2025-02-14 05:17:32,212 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26003.91 MB 2025-02-14 05:17:32,230 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8145, cut from 8147 2025-02-14 05:17:32,231 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 05:17:32,237 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 05:17:32,237 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 05:17:32,237 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:17:32,237 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:17:32,237 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18584.19 MB 2025-02-14 05:17:32,237 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27006.15 MB 2025-02-14 05:17:32,237 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8421.96 MB 2025-02-14 05:17:32,237 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26893.88 MB 2025-02-14 05:17:32,237 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37358.67 MB 2025-02-14 05:17:32,237 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10464.79 MB 2025-02-14 05:17:32,237 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27006.15 MB 2025-02-14 05:17:32,399 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7937] 2025-02-14 05:17:32,400 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:17:32,400 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 05:17:32,401 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:17:32,401 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 05:17:32,406 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 05:17:32,407 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:17:32,407 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 05:17:32,407 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 05:18:29,467 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:18:29,467 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 05:18:29,473 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 05:18:29,477 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:18:29,477 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 331, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 05:18:29,478 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:18:29,479 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 331, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 05:18:34,623 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 05:18:34,623 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 05:18:34,623 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.14 seconds 2025-02-14 05:18:34,623 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:18:34,623 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15275.17 MB 2025-02-14 05:18:34,623 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16447.48 MB 2025-02-14 05:18:34,623 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1172.31 MB 2025-02-14 05:18:34,623 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45730.50 MB 2025-02-14 05:18:34,623 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20222.84 MB 2025-02-14 05:18:34,623 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25507.66 MB 2025-02-14 05:18:34,623 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25426.82 MB 2025-02-14 05:18:34,637 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 05:18:34,637 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 05:18:34,637 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 05:18:34,637 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:18:34,637 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16447.48 MB 2025-02-14 05:18:34,637 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15082.76 MB 2025-02-14 05:18:34,637 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1364.71 MB 2025-02-14 05:18:34,637 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20222.84 MB 2025-02-14 05:18:34,637 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20222.84 MB 2025-02-14 05:18:34,637 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:18:34,637 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17273.05 MB 2025-02-14 05:18:34,909 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 05:18:34,909 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 05:18:34,909 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 05:18:34,909 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:18:34,909 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15082.76 MB 2025-02-14 05:18:34,909 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15157.08 MB 2025-02-14 05:18:34,909 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 74.32 MB 2025-02-14 05:18:34,909 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20222.84 MB 2025-02-14 05:18:34,909 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19050.53 MB 2025-02-14 05:18:34,909 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1172.31 MB 2025-02-14 05:18:34,909 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18656.89 MB 2025-02-14 05:18:34,914 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 05:18:34,914 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 05:18:34,914 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 05:18:34,914 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:18:34,914 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15157.01 MB 2025-02-14 05:18:34,914 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15421.49 MB 2025-02-14 05:18:34,914 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 264.47 MB 2025-02-14 05:18:34,914 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19050.53 MB 2025-02-14 05:18:34,914 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19050.53 MB 2025-02-14 05:18:34,914 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:18:34,914 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15619.93 MB 2025-02-14 05:18:34,969 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 05:18:34,969 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 05:18:34,969 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-14 05:18:34,969 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:18:34,969 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15421.49 MB 2025-02-14 05:18:34,969 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15742.74 MB 2025-02-14 05:18:34,969 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 321.25 MB 2025-02-14 05:18:34,969 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19050.53 MB 2025-02-14 05:18:34,969 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19050.53 MB 2025-02-14 05:18:34,969 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:18:34,969 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16511.54 MB 2025-02-14 05:18:34,969 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 05:18:34,969 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 05:18:34,969 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 05:18:34,969 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:18:34,969 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15157.01 MB 2025-02-14 05:18:34,969 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15742.74 MB 2025-02-14 05:18:34,969 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 585.73 MB 2025-02-14 05:18:34,970 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19050.53 MB 2025-02-14 05:18:34,970 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19050.53 MB 2025-02-14 05:18:34,970 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:18:34,970 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16511.54 MB 2025-02-14 05:18:34,999 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 05:18:34,999 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 05:18:34,999 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 05:18:34,999 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:18:34,999 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16053.38 MB 2025-02-14 05:18:34,999 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16188.29 MB 2025-02-14 05:18:34,999 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 134.91 MB 2025-02-14 05:18:34,999 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19050.53 MB 2025-02-14 05:18:34,999 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19130.22 MB 2025-02-14 05:18:34,999 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 79.69 MB 2025-02-14 05:18:34,999 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16287.38 MB 2025-02-14 05:18:35,014 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 05:18:35,014 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 05:18:35,014 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 05:18:35,014 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:18:35,014 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16273.63 MB 2025-02-14 05:18:35,014 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16410.07 MB 2025-02-14 05:18:35,014 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 136.44 MB 2025-02-14 05:18:35,014 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19130.22 MB 2025-02-14 05:18:35,014 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19132.32 MB 2025-02-14 05:18:35,014 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-14 05:18:35,014 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16410.07 MB 2025-02-14 05:18:35,015 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 05:18:35,015 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 05:18:35,015 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.53 seconds 2025-02-14 05:18:35,015 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:18:35,015 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14121.94 MB 2025-02-14 05:18:35,015 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16531.58 MB 2025-02-14 05:18:35,015 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2409.64 MB 2025-02-14 05:18:35,015 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45730.50 MB 2025-02-14 05:18:35,015 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19132.32 MB 2025-02-14 05:18:35,015 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26598.18 MB 2025-02-14 05:18:35,015 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16531.58 MB 2025-02-14 05:18:35,169 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 05:18:35,169 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 05:18:35,170 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 05:18:35,170 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:18:35,170 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16531.58 MB 2025-02-14 05:18:35,170 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16270.51 MB 2025-02-14 05:18:35,170 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -261.07 MB 2025-02-14 05:18:35,170 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19132.32 MB 2025-02-14 05:18:35,170 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19132.32 MB 2025-02-14 05:18:35,170 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:18:35,170 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18413.75 MB 2025-02-14 05:18:35,181 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 4927, cut from 4929 2025-02-14 05:18:35,181 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 05:18:35,185 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 05:18:35,185 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 05:18:35,185 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 05:18:35,185 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:18:35,185 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16270.51 MB 2025-02-14 05:18:35,185 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21370.66 MB 2025-02-14 05:18:35,185 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5100.15 MB 2025-02-14 05:18:35,185 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19132.32 MB 2025-02-14 05:18:35,185 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25472.01 MB 2025-02-14 05:18:35,185 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6339.69 MB 2025-02-14 05:18:35,185 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21370.66 MB 2025-02-14 05:18:35,283 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 4719] 2025-02-14 05:18:35,285 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:18:35,285 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 05:18:35,286 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:18:35,286 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 05:18:35,290 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 05:18:35,291 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:18:35,291 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 05:18:35,292 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 05:18:40,094 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:18:40,094 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 05:18:40,102 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 05:18:40,108 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:18:40,108 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1260, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 05:18:40,110 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:18:40,110 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1260, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 05:18:59,594 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 05:18:59,594 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 05:18:59,594 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.48 seconds 2025-02-14 05:18:59,594 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:18:59,594 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21748.59 MB 2025-02-14 05:18:59,594 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26207.66 MB 2025-02-14 05:18:59,594 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4459.07 MB 2025-02-14 05:18:59,594 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30542.92 MB 2025-02-14 05:18:59,594 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31511.81 MB 2025-02-14 05:18:59,594 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 968.88 MB 2025-02-14 05:18:59,594 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35071.14 MB 2025-02-14 05:18:59,675 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 05:18:59,675 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 05:18:59,675 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 05:18:59,675 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:18:59,675 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26207.66 MB 2025-02-14 05:18:59,675 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22328.20 MB 2025-02-14 05:18:59,675 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3879.46 MB 2025-02-14 05:18:59,675 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31511.81 MB 2025-02-14 05:18:59,675 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47133.49 MB 2025-02-14 05:18:59,675 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 15621.69 MB 2025-02-14 05:18:59,675 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39558.25 MB 2025-02-14 05:19:01,607 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 05:19:01,607 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 05:19:01,607 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 05:19:01,607 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:19:01,607 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22328.20 MB 2025-02-14 05:19:01,607 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22859.04 MB 2025-02-14 05:19:01,607 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 05:19:01,607 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47133.49 MB 2025-02-14 05:19:01,607 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25931.28 MB 2025-02-14 05:19:01,607 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21202.21 MB 2025-02-14 05:19:01,607 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26839.41 MB 2025-02-14 05:19:01,621 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 05:19:01,621 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 05:19:01,621 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 05:19:01,621 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:19:01,621 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22859.04 MB 2025-02-14 05:19:01,621 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24748.57 MB 2025-02-14 05:19:01,621 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 05:19:01,621 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25931.28 MB 2025-02-14 05:19:01,621 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29234.30 MB 2025-02-14 05:19:01,621 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 05:19:01,621 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26166.00 MB 2025-02-14 05:19:01,829 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 05:19:01,829 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 05:19:01,830 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 05:19:01,830 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:19:01,830 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24748.57 MB 2025-02-14 05:19:01,830 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26990.43 MB 2025-02-14 05:19:01,830 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 05:19:01,830 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29234.30 MB 2025-02-14 05:19:01,830 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35368.47 MB 2025-02-14 05:19:01,830 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 05:19:01,830 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32534.71 MB 2025-02-14 05:19:01,830 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 05:19:01,830 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 05:19:01,830 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 05:19:01,830 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:19:01,830 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22859.04 MB 2025-02-14 05:19:01,830 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26990.43 MB 2025-02-14 05:19:01,830 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 05:19:01,830 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25931.28 MB 2025-02-14 05:19:01,830 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35368.47 MB 2025-02-14 05:19:01,830 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9437.18 MB 2025-02-14 05:19:01,830 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32534.71 MB 2025-02-14 05:19:01,994 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 05:19:01,994 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 05:19:01,994 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 05:19:01,994 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:19:01,994 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28523.97 MB 2025-02-14 05:19:01,994 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29290.97 MB 2025-02-14 05:19:01,994 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 05:19:01,994 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35368.47 MB 2025-02-14 05:19:01,994 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35783.70 MB 2025-02-14 05:19:01,994 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 05:19:01,994 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29998.76 MB 2025-02-14 05:19:02,013 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 05:19:02,013 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 05:19:02,013 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:19:02,013 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:19:02,013 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29703.86 MB 2025-02-14 05:19:02,013 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29931.84 MB 2025-02-14 05:19:02,013 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.98 MB 2025-02-14 05:19:02,013 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35783.70 MB 2025-02-14 05:19:02,013 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35783.70 MB 2025-02-14 05:19:02,013 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:19:02,013 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30164.29 MB 2025-02-14 05:19:02,014 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 05:19:02,014 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 05:19:02,014 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.90 seconds 2025-02-14 05:19:02,014 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:19:02,014 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17358.65 MB 2025-02-14 05:19:02,014 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30131.73 MB 2025-02-14 05:19:02,014 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12773.08 MB 2025-02-14 05:19:02,014 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30542.92 MB 2025-02-14 05:19:02,014 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35783.70 MB 2025-02-14 05:19:02,014 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5240.78 MB 2025-02-14 05:19:02,014 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30164.29 MB 2025-02-14 05:19:02,281 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 05:19:02,281 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 05:19:02,281 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 05:19:02,281 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:19:02,281 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30131.73 MB 2025-02-14 05:19:02,281 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22344.75 MB 2025-02-14 05:19:02,281 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7786.98 MB 2025-02-14 05:19:02,281 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35783.70 MB 2025-02-14 05:19:02,281 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35783.70 MB 2025-02-14 05:19:02,281 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:19:02,281 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32628.65 MB 2025-02-14 05:19:02,299 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8114, cut from 8116 2025-02-14 05:19:02,299 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 05:19:02,305 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 05:19:02,305 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 05:19:02,305 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:19:02,305 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:19:02,305 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22344.75 MB 2025-02-14 05:19:02,305 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30733.90 MB 2025-02-14 05:19:02,305 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8389.15 MB 2025-02-14 05:19:02,305 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35783.70 MB 2025-02-14 05:19:02,305 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44126.18 MB 2025-02-14 05:19:02,305 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8342.47 MB 2025-02-14 05:19:02,305 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30733.90 MB 2025-02-14 05:19:02,467 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7906] 2025-02-14 05:19:02,468 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:19:02,468 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 05:19:02,469 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:19:02,469 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 05:19:02,474 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 05:19:02,475 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:19:02,475 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 05:19:02,475 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 05:21:07,503 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:21:07,504 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 05:21:07,509 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 05:21:07,513 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:21:07,513 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 61, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 05:21:07,514 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:21:07,515 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 61, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 05:21:08,466 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 05:21:08,466 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 05:21:08,466 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.95 seconds 2025-02-14 05:21:08,466 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:21:08,466 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13393.76 MB 2025-02-14 05:21:08,466 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 13609.64 MB 2025-02-14 05:21:08,466 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 215.88 MB 2025-02-14 05:21:08,466 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52468.65 MB 2025-02-14 05:21:08,466 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21980.25 MB 2025-02-14 05:21:08,466 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30488.40 MB 2025-02-14 05:21:08,466 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22205.29 MB 2025-02-14 05:21:08,469 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 05:21:08,469 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 05:21:08,469 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 05:21:08,469 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:21:08,469 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13609.64 MB 2025-02-14 05:21:08,469 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 13714.23 MB 2025-02-14 05:21:08,469 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 104.59 MB 2025-02-14 05:21:08,469 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21980.25 MB 2025-02-14 05:21:08,469 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21980.25 MB 2025-02-14 05:21:08,469 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:21:08,469 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14038.11 MB 2025-02-14 05:21:08,762 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 05:21:08,762 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 05:21:08,762 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-14 05:21:08,762 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:21:08,762 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13714.23 MB 2025-02-14 05:21:08,762 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 13795.18 MB 2025-02-14 05:21:08,762 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 80.95 MB 2025-02-14 05:21:08,762 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21980.25 MB 2025-02-14 05:21:08,762 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21980.25 MB 2025-02-14 05:21:08,762 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:21:08,762 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17607.48 MB 2025-02-14 05:21:08,767 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 05:21:08,767 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 05:21:08,767 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 05:21:08,767 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:21:08,767 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13795.12 MB 2025-02-14 05:21:08,767 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14083.20 MB 2025-02-14 05:21:08,767 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 288.08 MB 2025-02-14 05:21:08,767 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21980.25 MB 2025-02-14 05:21:08,767 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21980.25 MB 2025-02-14 05:21:08,767 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:21:08,767 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14299.37 MB 2025-02-14 05:21:08,828 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 05:21:08,828 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 05:21:08,828 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 05:21:08,828 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:21:08,828 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14083.20 MB 2025-02-14 05:21:08,828 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14433.14 MB 2025-02-14 05:21:08,828 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 349.93 MB 2025-02-14 05:21:08,828 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21980.25 MB 2025-02-14 05:21:08,828 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21980.25 MB 2025-02-14 05:21:08,828 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:21:08,828 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15270.59 MB 2025-02-14 05:21:08,828 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 05:21:08,828 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 05:21:08,829 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 05:21:08,829 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:21:08,829 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13795.18 MB 2025-02-14 05:21:08,829 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14433.14 MB 2025-02-14 05:21:08,829 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 637.95 MB 2025-02-14 05:21:08,829 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21980.25 MB 2025-02-14 05:21:08,829 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21980.25 MB 2025-02-14 05:21:08,829 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:21:08,829 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15270.59 MB 2025-02-14 05:21:08,863 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 05:21:08,863 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 05:21:08,863 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 05:21:08,864 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:21:08,864 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14770.94 MB 2025-02-14 05:21:08,864 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14917.89 MB 2025-02-14 05:21:08,864 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 146.95 MB 2025-02-14 05:21:08,864 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21980.25 MB 2025-02-14 05:21:08,864 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22070.43 MB 2025-02-14 05:21:08,864 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 90.18 MB 2025-02-14 05:21:08,864 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15025.83 MB 2025-02-14 05:21:08,868 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 05:21:08,868 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 05:21:08,868 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 05:21:08,868 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:21:08,868 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15010.85 MB 2025-02-14 05:21:08,868 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15157.95 MB 2025-02-14 05:21:08,868 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 147.10 MB 2025-02-14 05:21:08,868 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22070.43 MB 2025-02-14 05:21:08,868 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22070.43 MB 2025-02-14 05:21:08,868 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:21:08,868 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15157.95 MB 2025-02-14 05:21:08,869 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 05:21:08,869 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 05:21:08,869 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.35 seconds 2025-02-14 05:21:08,869 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:21:08,870 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13181.24 MB 2025-02-14 05:21:08,870 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15289.79 MB 2025-02-14 05:21:08,870 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2108.56 MB 2025-02-14 05:21:08,870 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52468.65 MB 2025-02-14 05:21:08,870 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22070.43 MB 2025-02-14 05:21:08,870 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30398.22 MB 2025-02-14 05:21:08,870 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15289.79 MB 2025-02-14 05:21:09,037 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 05:21:09,037 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 05:21:09,037 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 05:21:09,037 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:21:09,037 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15289.79 MB 2025-02-14 05:21:09,037 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15513.40 MB 2025-02-14 05:21:09,037 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 223.61 MB 2025-02-14 05:21:09,037 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22070.43 MB 2025-02-14 05:21:09,037 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22070.43 MB 2025-02-14 05:21:09,037 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:21:09,037 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17266.07 MB 2025-02-14 05:21:09,049 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 5347, cut from 5349 2025-02-14 05:21:09,049 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2 ('] 2025-02-14 05:21:09,054 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 05:21:09,054 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 05:21:09,054 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:21:09,054 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:21:09,054 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15513.40 MB 2025-02-14 05:21:09,054 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21046.61 MB 2025-02-14 05:21:09,054 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5533.21 MB 2025-02-14 05:21:09,054 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22070.43 MB 2025-02-14 05:21:09,054 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27573.35 MB 2025-02-14 05:21:09,054 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5502.93 MB 2025-02-14 05:21:09,054 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21046.61 MB 2025-02-14 05:21:09,162 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 5139] 2025-02-14 05:21:09,163 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:21:09,163 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 05:21:09,164 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:21:09,164 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 05:21:09,169 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 05:21:09,170 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:21:09,170 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 05:21:09,170 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2 ('] 2025-02-14 05:23:07,520 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:23:07,521 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 05:23:07,528 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 05:23:07,536 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:23:07,536 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 3208, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 05:23:07,538 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:23:07,538 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 3208, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 05:23:56,924 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 05:23:56,924 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 05:23:56,924 - resource_logging.py:150 - __exit__ - DEBUG - Time: 49.37 seconds 2025-02-14 05:23:56,924 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:23:56,924 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35324.35 MB 2025-02-14 05:23:56,924 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46678.33 MB 2025-02-14 05:23:56,924 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11353.98 MB 2025-02-14 05:23:56,924 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55431.92 MB 2025-02-14 05:23:56,924 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50616.86 MB 2025-02-14 05:23:56,924 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4815.06 MB 2025-02-14 05:23:56,924 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 58031.26 MB 2025-02-14 05:23:57,038 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 05:23:57,038 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 05:23:57,038 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 05:23:57,038 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:23:57,038 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46678.33 MB 2025-02-14 05:23:57,038 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32456.50 MB 2025-02-14 05:23:57,038 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -14221.83 MB 2025-02-14 05:23:57,038 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50616.86 MB 2025-02-14 05:23:57,038 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58191.77 MB 2025-02-14 05:23:57,038 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7574.91 MB 2025-02-14 05:23:57,038 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 55201.42 MB 2025-02-14 05:23:58,986 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 05:23:58,987 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 05:23:58,987 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.95 seconds 2025-02-14 05:23:58,987 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:23:58,987 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32456.50 MB 2025-02-14 05:23:58,987 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32987.34 MB 2025-02-14 05:23:58,987 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 05:23:58,987 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58191.77 MB 2025-02-14 05:23:58,987 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35001.47 MB 2025-02-14 05:23:58,987 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23190.31 MB 2025-02-14 05:23:58,987 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36967.71 MB 2025-02-14 05:23:59,001 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 05:23:59,001 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 05:23:59,001 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 05:23:59,001 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:23:59,001 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32987.34 MB 2025-02-14 05:23:59,001 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34876.88 MB 2025-02-14 05:23:59,001 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 05:23:59,001 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35001.47 MB 2025-02-14 05:23:59,001 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38304.48 MB 2025-02-14 05:23:59,001 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 05:23:59,001 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36294.30 MB 2025-02-14 05:23:59,211 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 05:23:59,211 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 05:23:59,211 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 05:23:59,211 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:23:59,211 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34876.88 MB 2025-02-14 05:23:59,211 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37118.73 MB 2025-02-14 05:23:59,211 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 05:23:59,211 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38304.48 MB 2025-02-14 05:23:59,211 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44910.51 MB 2025-02-14 05:23:59,211 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 05:23:59,211 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42663.01 MB 2025-02-14 05:23:59,212 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 05:23:59,212 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 05:23:59,212 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 05:23:59,212 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:23:59,212 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32987.34 MB 2025-02-14 05:23:59,212 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37118.73 MB 2025-02-14 05:23:59,212 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 05:23:59,212 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35001.47 MB 2025-02-14 05:23:59,212 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44910.51 MB 2025-02-14 05:23:59,212 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9909.04 MB 2025-02-14 05:23:59,212 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42663.01 MB 2025-02-14 05:23:59,380 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 05:23:59,380 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 05:23:59,380 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 05:23:59,380 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:23:59,380 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38652.27 MB 2025-02-14 05:23:59,380 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39419.28 MB 2025-02-14 05:23:59,380 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 05:23:59,380 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44910.51 MB 2025-02-14 05:23:59,380 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45327.84 MB 2025-02-14 05:23:59,380 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 05:23:59,380 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40127.06 MB 2025-02-14 05:23:59,398 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 05:23:59,398 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 05:23:59,398 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:23:59,398 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:23:59,398 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39832.16 MB 2025-02-14 05:23:59,398 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40060.12 MB 2025-02-14 05:23:59,398 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.95 MB 2025-02-14 05:23:59,398 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45327.84 MB 2025-02-14 05:23:59,398 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45327.84 MB 2025-02-14 05:23:59,398 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:23:59,398 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40259.06 MB 2025-02-14 05:23:59,400 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 05:23:59,400 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 05:23:59,400 - resource_logging.py:150 - __exit__ - DEBUG - Time: 51.86 seconds 2025-02-14 05:23:59,400 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:23:59,400 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24146.53 MB 2025-02-14 05:23:59,400 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40261.19 MB 2025-02-14 05:23:59,400 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 16114.66 MB 2025-02-14 05:23:59,400 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44254.10 MB 2025-02-14 05:23:59,400 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45327.84 MB 2025-02-14 05:23:59,400 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1073.74 MB 2025-02-14 05:23:59,400 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40261.19 MB 2025-02-14 05:23:59,671 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 05:23:59,671 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 05:23:59,671 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 05:23:59,671 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:23:59,671 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40261.19 MB 2025-02-14 05:23:59,671 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29150.92 MB 2025-02-14 05:23:59,671 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -11110.28 MB 2025-02-14 05:23:59,671 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45327.84 MB 2025-02-14 05:23:59,671 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45327.84 MB 2025-02-14 05:23:59,671 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:23:59,671 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42772.86 MB 2025-02-14 05:23:59,689 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 05:23:59,689 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 05:23:59,695 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 05:23:59,695 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 05:23:59,695 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:23:59,695 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:23:59,695 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29150.92 MB 2025-02-14 05:23:59,695 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37589.61 MB 2025-02-14 05:23:59,695 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8438.69 MB 2025-02-14 05:23:59,695 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45327.84 MB 2025-02-14 05:23:59,695 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49524.24 MB 2025-02-14 05:23:59,695 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4196.40 MB 2025-02-14 05:23:59,695 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37589.61 MB 2025-02-14 05:23:59,860 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 05:23:59,861 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:23:59,861 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 05:23:59,862 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:23:59,862 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 05:23:59,867 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 05:23:59,868 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:23:59,868 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 05:23:59,868 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 05:24:09,148 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:24:09,148 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 05:24:09,153 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 05:24:09,156 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:24:09,156 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2489, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 05:24:09,157 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:24:09,157 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2489, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 05:24:48,060 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 05:24:48,060 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 05:24:48,060 - resource_logging.py:150 - __exit__ - DEBUG - Time: 38.89 seconds 2025-02-14 05:24:48,060 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:24:48,060 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30312.50 MB 2025-02-14 05:24:48,060 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39120.93 MB 2025-02-14 05:24:48,060 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8808.43 MB 2025-02-14 05:24:48,060 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: [.56 MB 2025-02-14 05:24:48,060 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46665.83 MB 2025-02-14 05:24:48,060 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15514.73 MB 2025-02-14 05:24:48,060 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47938.41 MB 2025-02-14 05:24:48,216 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 05:24:48,216 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 05:24:48,216 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 05:24:48,216 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:24:48,216 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39120.93 MB 2025-02-14 05:24:48,216 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28717.43 MB 2025-02-14 05:24:48,216 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10403.50 MB 2025-02-14 05:24:48,216 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46665.83 MB 2025-02-14 05:24:48,216 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 75994.50 MB 2025-02-14 05:24:48,216 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 29328.67 MB 2025-02-14 05:24:48,216 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 64439.66 MB 2025-02-14 05:24:50,170 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 05:24:50,171 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 05:24:50,171 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.95 seconds 2025-02-14 05:24:50,171 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:24:50,171 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28717.43 MB 2025-02-14 05:24:50,171 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29248.27 MB 2025-02-14 05:24:50,171 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 05:24:50,171 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 75994.50 MB 2025-02-14 05:24:50,171 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33770.44 MB 2025-02-14 05:24:50,171 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -42224.06 MB 2025-02-14 05:24:50,171 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33227.61 MB 2025-02-14 05:24:50,184 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 05:24:50,184 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 05:24:50,184 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 05:24:50,184 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:24:50,184 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29248.27 MB 2025-02-14 05:24:50,184 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31137.81 MB 2025-02-14 05:24:50,184 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 05:24:50,184 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33770.44 MB 2025-02-14 05:24:50,184 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35657.88 MB 2025-02-14 05:24:50,184 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 05:24:50,184 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32555.23 MB 2025-02-14 05:24:50,393 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 05:24:50,393 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 05:24:50,393 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 05:24:50,393 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:24:50,393 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31137.81 MB 2025-02-14 05:24:50,393 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33379.66 MB 2025-02-14 05:24:50,393 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 05:24:50,393 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35657.88 MB 2025-02-14 05:24:50,393 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41792.05 MB 2025-02-14 05:24:50,393 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 05:24:50,393 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38923.94 MB 2025-02-14 05:24:50,393 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 05:24:50,393 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 05:24:50,393 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 05:24:50,393 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:24:50,393 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29248.27 MB 2025-02-14 05:24:50,394 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33379.66 MB 2025-02-14 05:24:50,394 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 05:24:50,394 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33770.44 MB 2025-02-14 05:24:50,394 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41792.05 MB 2025-02-14 05:24:50,394 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-14 05:24:50,394 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38923.94 MB 2025-02-14 05:24:50,577 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 05:24:50,577 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 05:24:50,577 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-14 05:24:50,577 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:24:50,577 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34913.20 MB 2025-02-14 05:24:50,577 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35680.21 MB 2025-02-14 05:24:50,577 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 05:24:50,577 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41792.05 MB 2025-02-14 05:24:50,577 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42207.28 MB 2025-02-14 05:24:50,577 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 05:24:50,577 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36387.99 MB 2025-02-14 05:24:50,596 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 05:24:50,596 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 05:24:50,596 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:24:50,596 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:24:50,596 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36093.09 MB 2025-02-14 05:24:50,596 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36321.39 MB 2025-02-14 05:24:50,596 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.30 MB 2025-02-14 05:24:50,596 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42207.28 MB 2025-02-14 05:24:50,596 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42207.28 MB 2025-02-14 05:24:50,596 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:24:50,596 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36555.59 MB 2025-02-14 05:24:50,597 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 05:24:50,597 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 05:24:50,597 - resource_logging.py:150 - __exit__ - DEBUG - Time: 41.44 seconds 2025-02-14 05:24:50,597 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:24:50,597 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21640.63 MB 2025-02-14 05:24:50,597 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36521.60 MB 2025-02-14 05:24:50,597 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14880.98 MB 2025-02-14 05:24:50,597 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62180.56 MB 2025-02-14 05:24:50,597 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42207.28 MB 2025-02-14 05:24:50,597 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19973.28 MB 2025-02-14 05:24:50,597 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36555.59 MB 2025-02-14 05:24:50,866 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 05:24:50,866 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 05:24:50,866 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 05:24:50,866 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:24:50,866 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36521.60 MB 2025-02-14 05:24:50,866 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26631.68 MB 2025-02-14 05:24:50,866 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9889.92 MB 2025-02-14 05:24:50,866 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42207.28 MB 2025-02-14 05:24:50,866 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42207.28 MB 2025-02-14 05:24:50,866 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:24:50,866 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39022.52 MB 2025-02-14 05:24:50,884 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8127, cut from 8129 2025-02-14 05:24:50,884 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2 ('] 2025-02-14 05:24:50,890 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 05:24:50,890 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 05:24:50,890 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:24:50,890 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:24:50,890 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26631.68 MB 2025-02-14 05:24:50,890 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35035.25 MB 2025-02-14 05:24:50,890 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8403.56 MB 2025-02-14 05:24:50,890 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42207.28 MB 2025-02-14 05:24:50,890 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46384.81 MB 2025-02-14 05:24:50,890 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4177.53 MB 2025-02-14 05:24:50,890 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35035.25 MB 2025-02-14 05:24:51,051 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7919] 2025-02-14 05:24:51,053 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:24:51,053 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 05:24:51,054 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:24:51,054 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 05:24:51,059 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 05:24:51,060 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:24:51,060 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 05:24:51,060 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2 ('] 2025-02-14 05:25:02,165 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:25:02,165 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 05:25:02,170 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 05:25:02,174 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:25:02,174 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 160, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 05:25:02,175 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:25:02,175 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 160, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 05:25:04,695 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 05:25:04,695 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 05:25:04,695 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.52 seconds 2025-02-14 05:25:04,695 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:25:04,695 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14083.61 MB 2025-02-14 05:25:04,695 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14649.84 MB 2025-02-14 05:25:04,695 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 566.23 MB 2025-02-14 05:25:04,695 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54739.86 MB 2025-02-14 05:25:04,695 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17727.23 MB 2025-02-14 05:25:04,695 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -37012.64 MB 2025-02-14 05:25:04,695 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23555.79 MB 2025-02-14 05:25:04,708 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 05:25:04,708 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 05:25:04,708 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 05:25:04,708 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:25:04,708 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14649.84 MB 2025-02-14 05:25:04,708 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14924.18 MB 2025-02-14 05:25:04,708 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 274.34 MB 2025-02-14 05:25:04,708 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17727.23 MB 2025-02-14 05:25:04,708 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18293.46 MB 2025-02-14 05:25:04,708 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 566.23 MB 2025-02-14 05:25:04,708 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16954.60 MB 2025-02-14 05:25:05,489 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 05:25:05,489 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 05:25:05,489 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.78 seconds 2025-02-14 05:25:05,489 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:25:05,489 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14924.18 MB 2025-02-14 05:25:05,489 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15136.52 MB 2025-02-14 05:25:05,489 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 212.34 MB 2025-02-14 05:25:05,489 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18293.46 MB 2025-02-14 05:25:05,489 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17876.12 MB 2025-02-14 05:25:05,489 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -417.33 MB 2025-02-14 05:25:05,489 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19095.65 MB 2025-02-14 05:25:05,497 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 05:25:05,497 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 05:25:05,497 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 05:25:05,497 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:25:05,497 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15136.45 MB 2025-02-14 05:25:05,497 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15892.08 MB 2025-02-14 05:25:05,497 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 755.63 MB 2025-02-14 05:25:05,497 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17876.12 MB 2025-02-14 05:25:05,497 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17876.12 MB 2025-02-14 05:25:05,497 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:25:05,497 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16459.06 MB 2025-02-14 05:25:05,584 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 05:25:05,585 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 05:25:05,585 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 05:25:05,585 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:25:05,585 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15892.08 MB 2025-02-14 05:25:05,585 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16788.86 MB 2025-02-14 05:25:05,585 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 896.78 MB 2025-02-14 05:25:05,585 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17876.12 MB 2025-02-14 05:25:05,585 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20141.05 MB 2025-02-14 05:25:05,585 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2264.92 MB 2025-02-14 05:25:05,585 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19006.54 MB 2025-02-14 05:25:05,585 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 05:25:05,585 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 05:25:05,585 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 05:25:05,585 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:25:05,585 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15136.45 MB 2025-02-14 05:25:05,585 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16788.86 MB 2025-02-14 05:25:05,585 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1652.41 MB 2025-02-14 05:25:05,585 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17876.12 MB 2025-02-14 05:25:05,585 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20141.05 MB 2025-02-14 05:25:05,585 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2264.92 MB 2025-02-14 05:25:05,585 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19006.54 MB 2025-02-14 05:25:05,653 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 05:25:05,653 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 05:25:05,653 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 05:25:05,653 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:25:05,653 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17402.28 MB 2025-02-14 05:25:05,653 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17709.08 MB 2025-02-14 05:25:05,653 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 306.80 MB 2025-02-14 05:25:05,653 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20141.05 MB 2025-02-14 05:25:05,653 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20306.72 MB 2025-02-14 05:25:05,653 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 165.68 MB 2025-02-14 05:25:05,653 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18000.71 MB 2025-02-14 05:25:05,663 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 05:25:05,663 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 05:25:05,663 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 05:25:05,663 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:25:05,663 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17874.24 MB 2025-02-14 05:25:05,663 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18102.55 MB 2025-02-14 05:25:05,663 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.31 MB 2025-02-14 05:25:05,663 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20306.72 MB 2025-02-14 05:25:05,663 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20306.72 MB 2025-02-14 05:25:05,663 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:25:05,663 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18124.82 MB 2025-02-14 05:25:05,664 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 05:25:05,664 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 05:25:05,664 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.49 seconds 2025-02-14 05:25:05,664 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:25:05,664 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13526.16 MB 2025-02-14 05:25:05,664 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18303.18 MB 2025-02-14 05:25:05,664 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4777.03 MB 2025-02-14 05:25:05,664 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54739.86 MB 2025-02-14 05:25:05,664 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20306.72 MB 2025-02-14 05:25:05,664 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34433.14 MB 2025-02-14 05:25:05,664 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18303.18 MB 2025-02-14 05:25:05,934 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 05:25:05,934 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 05:25:05,934 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 05:25:05,934 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:25:05,934 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18303.18 MB 2025-02-14 05:25:05,934 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17391.06 MB 2025-02-14 05:25:05,934 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -912.12 MB 2025-02-14 05:25:05,934 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20306.72 MB 2025-02-14 05:25:05,934 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20306.72 MB 2025-02-14 05:25:05,934 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:25:05,934 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19105.15 MB 2025-02-14 05:25:05,952 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8144, cut from 8146 2025-02-14 05:25:05,952 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2 ('] 2025-02-14 05:25:05,958 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 05:25:05,958 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 05:25:05,958 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:25:05,958 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:25:05,958 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17391.06 MB 2025-02-14 05:25:05,958 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25811.84 MB 2025-02-14 05:25:05,958 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8420.78 MB 2025-02-14 05:25:05,958 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20306.72 MB 2025-02-14 05:25:05,958 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30771.51 MB 2025-02-14 05:25:05,958 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10464.79 MB 2025-02-14 05:25:05,958 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25811.84 MB 2025-02-14 05:25:06,120 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7936] 2025-02-14 05:25:06,122 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:25:06,122 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 05:25:06,123 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:25:06,123 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 05:25:06,127 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 05:25:06,128 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:25:06,128 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 05:25:06,128 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2 ('] 2025-02-14 05:25:15,655 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:25:15,655 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 05:25:15,660 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 05:25:15,663 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:25:15,663 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 231, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 05:25:15,664 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:25:15,664 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 231, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 05:25:19,287 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 05:25:19,287 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 05:25:19,287 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.62 seconds 2025-02-14 05:25:19,287 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:25:19,287 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14578.35 MB 2025-02-14 05:25:19,287 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15395.85 MB 2025-02-14 05:25:19,287 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 817.50 MB 2025-02-14 05:25:19,287 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39143.34 MB 2025-02-14 05:25:19,287 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18471.71 MB 2025-02-14 05:25:19,287 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20671.63 MB 2025-02-14 05:25:19,287 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24277.02 MB 2025-02-14 05:25:19,297 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 05:25:19,297 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 05:25:19,297 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 05:25:19,297 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:25:19,297 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15395.85 MB 2025-02-14 05:25:19,297 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14274.95 MB 2025-02-14 05:25:19,297 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1120.90 MB 2025-02-14 05:25:19,297 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18471.71 MB 2025-02-14 05:25:19,297 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18471.71 MB 2025-02-14 05:25:19,297 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:25:19,297 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15631.38 MB 2025-02-14 05:25:19,383 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 05:25:19,383 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 05:25:19,383 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 05:25:19,383 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:25:19,383 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14274.95 MB 2025-02-14 05:25:19,383 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14294.86 MB 2025-02-14 05:25:19,383 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 19.91 MB 2025-02-14 05:25:19,383 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18471.71 MB 2025-02-14 05:25:19,383 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17473.47 MB 2025-02-14 05:25:19,383 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -998.24 MB 2025-02-14 05:25:19,383 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15232.31 MB 2025-02-14 05:25:19,387 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 05:25:19,387 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 05:25:19,387 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 05:25:19,387 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:25:19,387 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14294.79 MB 2025-02-14 05:25:19,387 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14365.63 MB 2025-02-14 05:25:19,387 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 70.84 MB 2025-02-14 05:25:19,387 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17473.47 MB 2025-02-14 05:25:19,387 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17473.47 MB 2025-02-14 05:25:19,387 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:25:19,387 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14418.79 MB 2025-02-14 05:25:19,400 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 05:25:19,401 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 05:25:19,401 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 05:25:19,401 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:25:19,401 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14365.63 MB 2025-02-14 05:25:19,401 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14449.76 MB 2025-02-14 05:25:19,401 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 84.13 MB 2025-02-14 05:25:19,401 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17473.47 MB 2025-02-14 05:25:19,401 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17473.47 MB 2025-02-14 05:25:19,401 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:25:19,401 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14657.61 MB 2025-02-14 05:25:19,401 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 05:25:19,401 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 05:25:19,401 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:25:19,401 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:25:19,401 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14294.79 MB 2025-02-14 05:25:19,401 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14449.76 MB 2025-02-14 05:25:19,401 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 154.97 MB 2025-02-14 05:25:19,401 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17473.47 MB 2025-02-14 05:25:19,401 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17473.47 MB 2025-02-14 05:25:19,401 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:25:19,401 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14657.61 MB 2025-02-14 05:25:19,410 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 05:25:19,410 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 05:25:19,410 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 05:25:19,410 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:25:19,410 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14507.27 MB 2025-02-14 05:25:19,410 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14536.03 MB 2025-02-14 05:25:19,410 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 28.76 MB 2025-02-14 05:25:19,410 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17473.47 MB 2025-02-14 05:25:19,410 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17486.05 MB 2025-02-14 05:25:19,410 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 12.58 MB 2025-02-14 05:25:19,410 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14572.67 MB 2025-02-14 05:25:19,413 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 05:25:19,413 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 05:25:19,413 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 05:25:19,413 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:25:19,413 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14551.53 MB 2025-02-14 05:25:19,413 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14571.42 MB 2025-02-14 05:25:19,413 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 19.90 MB 2025-02-14 05:25:19,413 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17486.05 MB 2025-02-14 05:25:19,413 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17486.05 MB 2025-02-14 05:25:19,413 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:25:19,413 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14571.42 MB 2025-02-14 05:25:19,414 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 05:25:19,414 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 05:25:19,414 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.75 seconds 2025-02-14 05:25:19,414 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:25:19,414 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13773.53 MB 2025-02-14 05:25:19,414 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14608.23 MB 2025-02-14 05:25:19,414 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 834.70 MB 2025-02-14 05:25:19,414 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39143.34 MB 2025-02-14 05:25:19,414 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17486.05 MB 2025-02-14 05:25:19,414 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21657.29 MB 2025-02-14 05:25:19,414 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14608.23 MB 2025-02-14 05:25:19,477 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 05:25:19,478 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 05:25:19,478 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 05:25:19,478 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:25:19,478 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14608.23 MB 2025-02-14 05:25:19,478 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15160.09 MB 2025-02-14 05:25:19,478 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 551.86 MB 2025-02-14 05:25:19,478 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17486.05 MB 2025-02-14 05:25:19,478 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17490.25 MB 2025-02-14 05:25:19,478 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-14 05:25:19,478 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15215.27 MB 2025-02-14 05:25:19,482 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 1483, cut from 1485 2025-02-14 05:25:19,482 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 05:25:19,484 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 05:25:19,484 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 05:25:19,484 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 05:25:19,484 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:25:19,484 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14416.07 MB 2025-02-14 05:25:19,484 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15960.75 MB 2025-02-14 05:25:19,484 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1544.68 MB 2025-02-14 05:25:19,484 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17490.25 MB 2025-02-14 05:25:19,484 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17490.25 MB 2025-02-14 05:25:19,484 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:25:19,484 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15960.75 MB 2025-02-14 05:25:19,514 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 1275] 2025-02-14 05:25:19,515 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:25:19,515 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 05:25:19,516 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:25:19,516 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 05:25:19,521 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 05:25:19,522 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:25:19,522 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 05:25:19,522 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 05:26:58,070 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:26:58,071 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 05:26:58,078 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 05:26:58,084 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:26:58,084 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 182, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 05:26:58,086 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:26:58,086 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 182, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 05:27:00,921 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 05:27:00,921 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 05:27:00,921 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.83 seconds 2025-02-14 05:27:00,921 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:27:00,921 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14236.91 MB 2025-02-14 05:27:00,921 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14881.00 MB 2025-02-14 05:27:00,921 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 644.09 MB 2025-02-14 05:27:00,921 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19813.89 MB 2025-02-14 05:27:00,921 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 16907.24 MB 2025-02-14 05:27:00,921 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2906.65 MB 2025-02-14 05:27:00,921 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23709.09 MB 2025-02-14 05:27:00,934 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 05:27:00,934 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 05:27:00,934 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 05:27:00,934 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:27:00,934 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14881.00 MB 2025-02-14 05:27:00,934 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15158.60 MB 2025-02-14 05:27:00,934 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 277.60 MB 2025-02-14 05:27:00,934 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 16907.24 MB 2025-02-14 05:27:00,934 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18465.42 MB 2025-02-14 05:27:00,934 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1558.18 MB 2025-02-14 05:27:00,934 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17390.36 MB 2025-02-14 05:27:01,793 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 05:27:01,793 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 05:27:01,793 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.86 seconds 2025-02-14 05:27:01,793 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:27:01,793 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15158.60 MB 2025-02-14 05:27:01,793 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15393.50 MB 2025-02-14 05:27:01,793 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 234.90 MB 2025-02-14 05:27:01,793 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18465.42 MB 2025-02-14 05:27:01,793 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17842.57 MB 2025-02-14 05:27:01,793 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -622.85 MB 2025-02-14 05:27:01,793 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19330.07 MB 2025-02-14 05:27:01,801 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 05:27:01,801 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 05:27:01,801 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 05:27:01,801 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:27:01,801 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15393.43 MB 2025-02-14 05:27:01,801 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16229.35 MB 2025-02-14 05:27:01,801 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 835.92 MB 2025-02-14 05:27:01,801 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17842.57 MB 2025-02-14 05:27:01,801 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18262.00 MB 2025-02-14 05:27:01,801 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 419.43 MB 2025-02-14 05:27:01,801 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16856.56 MB 2025-02-14 05:27:01,896 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 05:27:01,896 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 05:27:01,896 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 05:27:01,896 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:27:01,897 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16229.35 MB 2025-02-14 05:27:01,897 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17221.40 MB 2025-02-14 05:27:01,897 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 992.06 MB 2025-02-14 05:27:01,897 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18262.00 MB 2025-02-14 05:27:01,897 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20778.58 MB 2025-02-14 05:27:01,897 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2516.58 MB 2025-02-14 05:27:01,897 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19674.71 MB 2025-02-14 05:27:01,897 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 05:27:01,897 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 05:27:01,897 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 05:27:01,897 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:27:01,897 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15393.43 MB 2025-02-14 05:27:01,897 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17221.40 MB 2025-02-14 05:27:01,897 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1827.97 MB 2025-02-14 05:27:01,897 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17842.57 MB 2025-02-14 05:27:01,897 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20778.58 MB 2025-02-14 05:27:01,897 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2936.01 MB 2025-02-14 05:27:01,897 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19674.71 MB 2025-02-14 05:27:01,972 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 05:27:01,972 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 05:27:01,972 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 05:27:01,972 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:27:01,972 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17900.00 MB 2025-02-14 05:27:01,972 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18239.40 MB 2025-02-14 05:27:01,972 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 339.40 MB 2025-02-14 05:27:01,972 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20778.58 MB 2025-02-14 05:27:01,972 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20961.03 MB 2025-02-14 05:27:01,972 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 182.45 MB 2025-02-14 05:27:01,972 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18559.64 MB 2025-02-14 05:27:01,982 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 05:27:01,982 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 05:27:01,982 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 05:27:01,982 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:27:01,982 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18422.28 MB 2025-02-14 05:27:01,982 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18651.93 MB 2025-02-14 05:27:01,982 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.64 MB 2025-02-14 05:27:01,982 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20961.03 MB 2025-02-14 05:27:01,982 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20961.03 MB 2025-02-14 05:27:01,982 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:27:01,982 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18694.22 MB 2025-02-14 05:27:01,983 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 05:27:01,983 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 05:27:01,983 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.89 seconds 2025-02-14 05:27:01,983 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:27:01,983 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13602.81 MB 2025-02-14 05:27:01,983 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18853.00 MB 2025-02-14 05:27:01,983 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5250.19 MB 2025-02-14 05:27:01,983 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19813.89 MB 2025-02-14 05:27:01,983 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20961.03 MB 2025-02-14 05:27:01,983 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1147.14 MB 2025-02-14 05:27:01,983 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18853.00 MB 2025-02-14 05:27:02,248 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 05:27:02,248 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 05:27:02,248 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 05:27:02,248 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:27:02,248 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18853.00 MB 2025-02-14 05:27:02,248 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17554.53 MB 2025-02-14 05:27:02,248 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1298.47 MB 2025-02-14 05:27:02,248 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20961.03 MB 2025-02-14 05:27:02,248 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20961.03 MB 2025-02-14 05:27:02,248 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:27:02,248 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19087.42 MB 2025-02-14 05:27:02,266 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 05:27:02,266 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 video rate for this video is 2,'] 2025-02-14 05:27:02,272 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 05:27:02,272 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 05:27:02,272 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:27:02,272 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:27:02,272 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17554.53 MB 2025-02-14 05:27:02,272 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25993.56 MB 2025-02-14 05:27:02,272 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 05:27:02,272 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20961.03 MB 2025-02-14 05:27:02,272 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31450.99 MB 2025-02-14 05:27:02,272 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 05:27:02,272 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25993.56 MB 2025-02-14 05:27:02,434 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 05:27:02,435 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:27:02,435 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 05:27:02,436 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:27:02,436 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 05:27:02,441 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 05:27:02,442 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:27:02,442 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 05:27:02,442 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 video rate for this video is 2,'] 2025-02-14 05:28:52,722 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:28:52,723 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 05:28:52,731 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 05:28:52,738 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:28:52,738 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2099, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 05:28:52,740 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:28:52,740 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2099, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 05:29:24,968 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 05:29:24,969 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 05:29:24,969 - resource_logging.py:150 - __exit__ - DEBUG - Time: 32.22 seconds 2025-02-14 05:29:24,969 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:29:24,969 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27594.87 MB 2025-02-14 05:29:24,969 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35023.12 MB 2025-02-14 05:29:24,969 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7428.24 MB 2025-02-14 05:29:24,969 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44036.00 MB 2025-02-14 05:29:24,969 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41116.76 MB 2025-02-14 05:29:24,969 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2919.24 MB 2025-02-14 05:29:24,969 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43862.75 MB 2025-02-14 05:29:25,107 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 05:29:25,108 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 05:29:25,108 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-14 05:29:25,108 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:29:25,108 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35023.12 MB 2025-02-14 05:29:25,108 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26689.90 MB 2025-02-14 05:29:25,108 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8333.22 MB 2025-02-14 05:29:25,108 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41116.76 MB 2025-02-14 05:29:25,108 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65582.14 MB 2025-02-14 05:29:25,108 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 24465.38 MB 2025-02-14 05:29:25,108 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 55579.45 MB 2025-02-14 05:29:27,048 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 05:29:27,048 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 05:29:27,048 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-14 05:29:27,048 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:29:27,048 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26689.90 MB 2025-02-14 05:29:27,048 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27220.74 MB 2025-02-14 05:29:27,048 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 05:29:27,048 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65582.14 MB 2025-02-14 05:29:27,048 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30907.83 MB 2025-02-14 05:29:27,048 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34674.31 MB 2025-02-14 05:29:27,048 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31201.11 MB 2025-02-14 05:29:27,062 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 05:29:27,062 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 05:29:27,062 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 05:29:27,062 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:29:27,062 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27220.74 MB 2025-02-14 05:29:27,062 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29110.27 MB 2025-02-14 05:29:27,062 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 05:29:27,062 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30907.83 MB 2025-02-14 05:29:27,062 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33738.98 MB 2025-02-14 05:29:27,062 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-14 05:29:27,062 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30527.70 MB 2025-02-14 05:29:27,271 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 05:29:27,271 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 05:29:27,271 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 05:29:27,271 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:29:27,271 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29110.27 MB 2025-02-14 05:29:27,271 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31352.13 MB 2025-02-14 05:29:27,271 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 05:29:27,271 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33738.98 MB 2025-02-14 05:29:27,271 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39401.29 MB 2025-02-14 05:29:27,272 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 05:29:27,272 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36896.41 MB 2025-02-14 05:29:27,272 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 05:29:27,272 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 05:29:27,273 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 05:29:27,273 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:29:27,273 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27220.74 MB 2025-02-14 05:29:27,273 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31352.13 MB 2025-02-14 05:29:27,273 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 05:29:27,273 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30907.83 MB 2025-02-14 05:29:27,273 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39401.29 MB 2025-02-14 05:29:27,273 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8493.47 MB 2025-02-14 05:29:27,273 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36896.41 MB 2025-02-14 05:29:27,440 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 05:29:27,440 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 05:29:27,441 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 05:29:27,441 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:29:27,441 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32885.67 MB 2025-02-14 05:29:27,441 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33652.67 MB 2025-02-14 05:29:27,441 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 05:29:27,441 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39401.29 MB 2025-02-14 05:29:27,441 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39818.63 MB 2025-02-14 05:29:27,441 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 05:29:27,441 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34360.46 MB 2025-02-14 05:29:27,459 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 05:29:27,459 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 05:29:27,459 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:29:27,459 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:29:27,459 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34065.56 MB 2025-02-14 05:29:27,459 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34294.40 MB 2025-02-14 05:29:27,459 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.84 MB 2025-02-14 05:29:27,459 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39818.63 MB 2025-02-14 05:29:27,459 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39818.63 MB 2025-02-14 05:29:27,459 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:29:27,459 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34519.82 MB 2025-02-14 05:29:27,460 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 05:29:27,460 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 05:29:27,460 - resource_logging.py:150 - __exit__ - DEBUG - Time: 34.72 seconds 2025-02-14 05:29:27,460 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:29:27,460 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20281.79 MB 2025-02-14 05:29:27,460 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34494.93 MB 2025-02-14 05:29:27,460 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14213.14 MB 2025-02-14 05:29:27,460 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44036.00 MB 2025-02-14 05:29:27,460 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39818.63 MB 2025-02-14 05:29:27,460 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4217.37 MB 2025-02-14 05:29:27,460 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34519.82 MB 2025-02-14 05:29:27,728 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 05:29:27,728 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 05:29:27,728 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 05:29:27,728 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:29:27,728 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34494.93 MB 2025-02-14 05:29:27,728 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25277.80 MB 2025-02-14 05:29:27,728 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9217.13 MB 2025-02-14 05:29:27,728 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39818.63 MB 2025-02-14 05:29:27,728 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39818.63 MB 2025-02-14 05:29:27,728 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:29:27,728 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36999.84 MB 2025-02-14 05:29:27,746 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8140, cut from 8142 2025-02-14 05:29:27,747 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2 ('] 2025-02-14 05:29:27,753 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 05:29:27,753 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 05:29:27,753 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:29:27,753 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:29:27,753 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25277.80 MB 2025-02-14 05:29:27,753 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33694.40 MB 2025-02-14 05:29:27,753 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8416.60 MB 2025-02-14 05:29:27,753 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39818.63 MB 2025-02-14 05:29:27,753 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48186.26 MB 2025-02-14 05:29:27,753 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8367.64 MB 2025-02-14 05:29:27,753 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33694.40 MB 2025-02-14 05:29:27,916 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7932] 2025-02-14 05:29:27,917 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:29:27,917 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 05:29:27,918 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:29:27,918 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 05:29:27,923 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 05:29:27,924 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:29:27,924 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 05:29:27,924 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2 ('] 2025-02-14 05:30:39,290 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:30:39,291 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 05:30:39,296 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 05:30:39,299 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:30:39,299 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2332, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 05:30:39,300 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:30:39,300 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2332, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 05:31:15,381 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 05:31:15,381 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 05:31:15,381 - resource_logging.py:150 - __exit__ - DEBUG - Time: 36.07 seconds 2025-02-14 05:31:15,381 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:31:15,381 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29218.46 MB 2025-02-14 05:31:15,381 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37471.27 MB 2025-02-14 05:31:15,381 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8252.82 MB 2025-02-14 05:31:15,381 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56553.90 MB 2025-02-14 05:31:15,381 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41899.00 MB 2025-02-14 05:31:15,381 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14654.90 MB 2025-02-14 05:31:15,381 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46391.38 MB 2025-02-14 05:31:15,528 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 05:31:15,528 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 05:31:15,528 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 05:31:15,528 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:31:15,528 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37471.27 MB 2025-02-14 05:31:15,528 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27901.19 MB 2025-02-14 05:31:15,528 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9570.08 MB 2025-02-14 05:31:15,528 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41899.00 MB 2025-02-14 05:31:15,528 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 71082.97 MB 2025-02-14 05:31:15,528 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 29183.97 MB 2025-02-14 05:31:15,528 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 60281.86 MB 2025-02-14 05:31:17,494 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 05:31:17,494 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 05:31:17,494 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.96 seconds 2025-02-14 05:31:17,494 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:31:17,494 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27901.19 MB 2025-02-14 05:31:17,494 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28432.03 MB 2025-02-14 05:31:17,494 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 05:31:17,494 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 71082.97 MB 2025-02-14 05:31:17,494 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30876.37 MB 2025-02-14 05:31:17,494 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -40206.60 MB 2025-02-14 05:31:17,494 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32412.40 MB 2025-02-14 05:31:17,508 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 05:31:17,508 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 05:31:17,508 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 05:31:17,508 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:31:17,508 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28432.03 MB 2025-02-14 05:31:17,508 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30321.57 MB 2025-02-14 05:31:17,508 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 05:31:17,508 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30876.37 MB 2025-02-14 05:31:17,508 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34179.38 MB 2025-02-14 05:31:17,508 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 05:31:17,508 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31739.00 MB 2025-02-14 05:31:17,717 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 05:31:17,717 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 05:31:17,717 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 05:31:17,717 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:31:17,717 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30321.57 MB 2025-02-14 05:31:17,717 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32563.42 MB 2025-02-14 05:31:17,717 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 05:31:17,717 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34179.38 MB 2025-02-14 05:31:17,717 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40785.41 MB 2025-02-14 05:31:17,717 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 05:31:17,718 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38107.70 MB 2025-02-14 05:31:17,718 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 05:31:17,718 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 05:31:17,718 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 05:31:17,718 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:31:17,718 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28432.03 MB 2025-02-14 05:31:17,718 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32563.42 MB 2025-02-14 05:31:17,718 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 05:31:17,718 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30876.37 MB 2025-02-14 05:31:17,718 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40785.41 MB 2025-02-14 05:31:17,718 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9909.04 MB 2025-02-14 05:31:17,718 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38107.70 MB 2025-02-14 05:31:17,884 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 05:31:17,884 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 05:31:17,884 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 05:31:17,884 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:31:17,884 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34096.97 MB 2025-02-14 05:31:17,884 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34863.97 MB 2025-02-14 05:31:17,884 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 05:31:17,884 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40785.41 MB 2025-02-14 05:31:17,884 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41200.65 MB 2025-02-14 05:31:17,884 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 05:31:17,884 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35571.76 MB 2025-02-14 05:31:17,902 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 05:31:17,903 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 05:31:17,903 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:31:17,903 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:31:17,903 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35276.86 MB 2025-02-14 05:31:17,903 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35506.01 MB 2025-02-14 05:31:17,903 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.16 MB 2025-02-14 05:31:17,903 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41200.65 MB 2025-02-14 05:31:17,903 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41200.65 MB 2025-02-14 05:31:17,903 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:31:17,903 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35718.63 MB 2025-02-14 05:31:17,904 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 05:31:17,904 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 05:31:17,904 - resource_logging.py:150 - __exit__ - DEBUG - Time: 38.60 seconds 2025-02-14 05:31:17,904 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:31:17,904 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21093.58 MB 2025-02-14 05:31:17,904 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35707.06 MB 2025-02-14 05:31:17,904 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14613.48 MB 2025-02-14 05:31:17,904 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56553.90 MB 2025-02-14 05:31:17,904 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41200.65 MB 2025-02-14 05:31:17,904 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15353.25 MB 2025-02-14 05:31:17,904 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35718.63 MB 2025-02-14 05:31:18,173 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 05:31:18,173 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 05:31:18,173 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 05:31:18,173 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:31:18,173 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35707.06 MB 2025-02-14 05:31:18,173 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26097.59 MB 2025-02-14 05:31:18,173 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9609.47 MB 2025-02-14 05:31:18,173 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41200.65 MB 2025-02-14 05:31:18,173 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41200.65 MB 2025-02-14 05:31:18,173 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:31:18,173 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38218.42 MB 2025-02-14 05:31:18,191 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8161, cut from 8163 2025-02-14 05:31:18,192 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 05:31:18,198 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 05:31:18,198 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 05:31:18,198 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:31:18,198 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:31:18,198 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26097.59 MB 2025-02-14 05:31:18,198 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34536.43 MB 2025-02-14 05:31:18,198 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8438.84 MB 2025-02-14 05:31:18,198 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41200.65 MB 2025-02-14 05:31:18,198 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49589.26 MB 2025-02-14 05:31:18,198 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8388.61 MB 2025-02-14 05:31:18,198 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34536.43 MB 2025-02-14 05:31:18,365 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7953] 2025-02-14 05:31:18,366 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:31:18,366 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 05:31:18,367 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:31:18,367 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 05:31:18,372 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 05:31:18,374 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:31:18,374 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 05:31:18,374 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 05:32:22,796 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:32:22,797 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 05:32:22,805 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 05:32:22,812 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:32:22,813 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1482, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 05:32:22,814 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:32:22,814 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1482, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 05:32:45,753 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 05:32:45,753 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 05:32:45,753 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.93 seconds 2025-02-14 05:32:45,753 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:32:45,753 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23295.52 MB 2025-02-14 05:32:45,753 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28540.50 MB 2025-02-14 05:32:45,753 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5244.98 MB 2025-02-14 05:32:45,753 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57977.86 MB 2025-02-14 05:32:45,753 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38900.07 MB 2025-02-14 05:32:45,753 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19077.79 MB 2025-02-14 05:32:45,753 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37523.23 MB 2025-02-14 05:32:45,835 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 05:32:45,835 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 05:32:45,835 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 05:32:45,835 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:32:45,835 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28540.50 MB 2025-02-14 05:32:45,835 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23482.31 MB 2025-02-14 05:32:45,835 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5058.19 MB 2025-02-14 05:32:45,835 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38900.07 MB 2025-02-14 05:32:45,835 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48727.33 MB 2025-02-14 05:32:45,836 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9827.25 MB 2025-02-14 05:32:45,836 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43352.93 MB 2025-02-14 05:32:47,760 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 05:32:47,760 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 05:32:47,760 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 05:32:47,760 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:32:47,760 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23482.31 MB 2025-02-14 05:32:47,760 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24013.15 MB 2025-02-14 05:32:47,760 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 05:32:47,760 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48727.33 MB 2025-02-14 05:32:47,760 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29471.28 MB 2025-02-14 05:32:47,760 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19256.05 MB 2025-02-14 05:32:47,760 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27992.48 MB 2025-02-14 05:32:47,775 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 05:32:47,775 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 05:32:47,775 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 05:32:47,775 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:32:47,775 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24013.15 MB 2025-02-14 05:32:47,776 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25902.68 MB 2025-02-14 05:32:47,776 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 05:32:47,776 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29471.28 MB 2025-02-14 05:32:47,776 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30415.00 MB 2025-02-14 05:32:47,776 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 05:32:47,776 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27320.11 MB 2025-02-14 05:32:47,988 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 05:32:47,988 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 05:32:47,988 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 05:32:47,988 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:32:47,988 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25902.68 MB 2025-02-14 05:32:47,988 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28144.54 MB 2025-02-14 05:32:47,988 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 05:32:47,988 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30415.00 MB 2025-02-14 05:32:47,988 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36077.31 MB 2025-02-14 05:32:47,988 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 05:32:47,988 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33688.82 MB 2025-02-14 05:32:47,989 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 05:32:47,989 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 05:32:47,989 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 05:32:47,989 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:32:47,989 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24013.15 MB 2025-02-14 05:32:47,989 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28144.54 MB 2025-02-14 05:32:47,989 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 05:32:47,989 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29471.28 MB 2025-02-14 05:32:47,989 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36077.31 MB 2025-02-14 05:32:47,989 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 05:32:47,989 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33688.82 MB 2025-02-14 05:32:48,161 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 05:32:48,161 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 05:32:48,161 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 05:32:48,161 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:32:48,161 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29678.08 MB 2025-02-14 05:32:48,161 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30445.08 MB 2025-02-14 05:32:48,161 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 05:32:48,161 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36077.31 MB 2025-02-14 05:32:48,161 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36492.54 MB 2025-02-14 05:32:48,161 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 05:32:48,161 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31152.87 MB 2025-02-14 05:32:48,179 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 05:32:48,179 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 05:32:48,180 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:32:48,180 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:32:48,180 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30857.97 MB 2025-02-14 05:32:48,180 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31086.07 MB 2025-02-14 05:32:48,180 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.10 MB 2025-02-14 05:32:48,180 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36492.54 MB 2025-02-14 05:32:48,180 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36492.54 MB 2025-02-14 05:32:48,180 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:32:48,180 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31286.72 MB 2025-02-14 05:32:48,181 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 05:32:48,181 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 05:32:48,181 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.36 seconds 2025-02-14 05:32:48,181 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:32:48,181 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18132.11 MB 2025-02-14 05:32:48,181 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31286.21 MB 2025-02-14 05:32:48,181 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13154.10 MB 2025-02-14 05:32:48,181 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57977.86 MB 2025-02-14 05:32:48,181 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36492.54 MB 2025-02-14 05:32:48,181 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21485.32 MB 2025-02-14 05:32:48,181 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31286.72 MB 2025-02-14 05:32:48,450 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 05:32:48,450 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 05:32:48,450 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 05:32:48,450 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:32:48,450 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31286.21 MB 2025-02-14 05:32:48,450 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23122.03 MB 2025-02-14 05:32:48,450 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8164.18 MB 2025-02-14 05:32:48,450 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36492.54 MB 2025-02-14 05:32:48,450 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36492.54 MB 2025-02-14 05:32:48,450 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:32:48,450 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33786.20 MB 2025-02-14 05:32:48,468 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8124, cut from 8126 2025-02-14 05:32:48,468 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-14 05:32:48,474 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 05:32:48,474 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 05:32:48,474 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:32:48,474 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:32:48,474 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23122.03 MB 2025-02-14 05:32:48,474 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31522.89 MB 2025-02-14 05:32:48,474 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8400.86 MB 2025-02-14 05:32:48,474 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36492.54 MB 2025-02-14 05:32:48,474 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44843.40 MB 2025-02-14 05:32:48,475 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8350.86 MB 2025-02-14 05:32:48,475 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31522.89 MB 2025-02-14 05:32:48,642 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7916] 2025-02-14 05:32:48,643 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:32:48,644 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 05:32:48,644 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:32:48,645 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 05:32:48,649 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 05:32:48,650 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:32:48,651 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 05:32:48,651 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-14 05:32:56,995 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:32:56,995 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 05:32:57,000 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 05:32:57,004 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:32:57,004 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1510, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 05:32:57,005 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:32:57,005 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1510, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 05:33:20,517 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 05:33:20,517 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 05:33:20,517 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.51 seconds 2025-02-14 05:33:20,517 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:33:20,517 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23490.63 MB 2025-02-14 05:33:20,517 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28834.43 MB 2025-02-14 05:33:20,517 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5343.81 MB 2025-02-14 05:33:20,517 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53194.26 MB 2025-02-14 05:33:20,517 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38962.99 MB 2025-02-14 05:33:20,517 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14231.27 MB 2025-02-14 05:33:20,517 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37719.15 MB 2025-02-14 05:33:20,586 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 05:33:20,587 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 05:33:20,587 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 05:33:20,587 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:33:20,587 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28834.43 MB 2025-02-14 05:33:20,587 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23627.87 MB 2025-02-14 05:33:20,587 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5206.56 MB 2025-02-14 05:33:20,587 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38962.99 MB 2025-02-14 05:33:20,587 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42070.97 MB 2025-02-14 05:33:20,587 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3107.98 MB 2025-02-14 05:33:20,587 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39247.46 MB 2025-02-14 05:33:22,514 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 05:33:22,514 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 05:33:22,514 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 05:33:22,514 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:33:22,514 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23627.87 MB 2025-02-14 05:33:22,514 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24158.71 MB 2025-02-14 05:33:22,514 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 05:33:22,514 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42070.97 MB 2025-02-14 05:33:22,514 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29441.92 MB 2025-02-14 05:33:22,514 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12629.05 MB 2025-02-14 05:33:22,514 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28138.05 MB 2025-02-14 05:33:22,529 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 05:33:22,529 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 05:33:22,529 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 05:33:22,529 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:33:22,529 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24158.71 MB 2025-02-14 05:33:22,529 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26048.25 MB 2025-02-14 05:33:22,529 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 05:33:22,529 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29441.92 MB 2025-02-14 05:33:22,529 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30385.64 MB 2025-02-14 05:33:22,529 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 05:33:22,529 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27465.67 MB 2025-02-14 05:33:22,738 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 05:33:22,738 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 05:33:22,738 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 05:33:22,738 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:33:22,738 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26048.25 MB 2025-02-14 05:33:22,738 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28290.10 MB 2025-02-14 05:33:22,738 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 05:33:22,738 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30385.64 MB 2025-02-14 05:33:22,738 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36047.95 MB 2025-02-14 05:33:22,738 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 05:33:22,738 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33834.38 MB 2025-02-14 05:33:22,738 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 05:33:22,738 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 05:33:22,738 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 05:33:22,738 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:33:22,738 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24158.71 MB 2025-02-14 05:33:22,738 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28290.10 MB 2025-02-14 05:33:22,738 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 05:33:22,738 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29441.92 MB 2025-02-14 05:33:22,739 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36047.95 MB 2025-02-14 05:33:22,739 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 05:33:22,739 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33834.38 MB 2025-02-14 05:33:22,903 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 05:33:22,903 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 05:33:22,903 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 05:33:22,903 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:33:22,903 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29823.64 MB 2025-02-14 05:33:22,903 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30590.65 MB 2025-02-14 05:33:22,903 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 05:33:22,903 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36047.95 MB 2025-02-14 05:33:22,903 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36465.28 MB 2025-02-14 05:33:22,903 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 05:33:22,903 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31298.43 MB 2025-02-14 05:33:22,922 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 05:33:22,922 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 05:33:22,922 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:33:22,922 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:33:22,922 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31003.53 MB 2025-02-14 05:33:22,922 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31232.63 MB 2025-02-14 05:33:22,922 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.10 MB 2025-02-14 05:33:22,922 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36465.28 MB 2025-02-14 05:33:22,922 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36465.28 MB 2025-02-14 05:33:22,922 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:33:22,922 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31431.09 MB 2025-02-14 05:33:22,923 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 05:33:22,923 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 05:33:22,923 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.92 seconds 2025-02-14 05:33:22,923 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:33:22,923 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18229.67 MB 2025-02-14 05:33:22,923 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31433.70 MB 2025-02-14 05:33:22,923 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13204.04 MB 2025-02-14 05:33:22,923 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53194.26 MB 2025-02-14 05:33:22,923 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36465.28 MB 2025-02-14 05:33:22,923 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16728.98 MB 2025-02-14 05:33:22,923 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31433.70 MB 2025-02-14 05:33:23,196 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 05:33:23,196 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 05:33:23,196 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 05:33:23,196 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:33:23,196 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31433.70 MB 2025-02-14 05:33:23,196 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23234.06 MB 2025-02-14 05:33:23,196 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8199.65 MB 2025-02-14 05:33:23,196 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36465.28 MB 2025-02-14 05:33:23,196 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36465.28 MB 2025-02-14 05:33:23,196 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:33:23,196 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33945.37 MB 2025-02-14 05:33:23,214 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 05:33:23,215 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 05:33:23,221 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 05:33:23,221 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 05:33:23,221 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:33:23,221 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:33:23,221 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23234.06 MB 2025-02-14 05:33:23,221 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31673.08 MB 2025-02-14 05:33:23,221 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 05:33:23,221 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36465.28 MB 2025-02-14 05:33:23,221 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44855.98 MB 2025-02-14 05:33:23,221 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 05:33:23,221 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31673.08 MB 2025-02-14 05:33:23,385 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 05:33:23,387 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:33:23,387 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 05:33:23,388 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:33:23,388 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 05:33:23,392 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 05:33:23,393 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:33:23,393 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 05:33:23,394 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 05:34:20,628 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:34:20,629 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 05:34:20,634 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 05:34:20,638 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:34:20,638 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 145, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 05:34:20,639 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:34:20,639 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 145, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 05:34:22,906 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 05:34:22,906 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 05:34:22,906 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.26 seconds 2025-02-14 05:34:22,906 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:34:22,906 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13979.09 MB 2025-02-14 05:34:22,906 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14492.24 MB 2025-02-14 05:34:22,906 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 513.15 MB 2025-02-14 05:34:22,906 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57440.99 MB 2025-02-14 05:34:22,906 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 16909.34 MB 2025-02-14 05:34:22,906 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -40531.66 MB 2025-02-14 05:34:22,906 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23451.27 MB 2025-02-14 05:34:22,915 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 05:34:22,915 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 05:34:22,915 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 05:34:22,915 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:34:22,915 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14492.24 MB 2025-02-14 05:34:22,915 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14150.92 MB 2025-02-14 05:34:22,915 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -341.32 MB 2025-02-14 05:34:22,915 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 16909.34 MB 2025-02-14 05:34:22,915 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 16909.34 MB 2025-02-14 05:34:22,915 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:34:22,915 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15352.65 MB 2025-02-14 05:34:23,208 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 05:34:23,208 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 05:34:23,208 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-14 05:34:23,208 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:34:23,208 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14150.92 MB 2025-02-14 05:34:23,208 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14231.87 MB 2025-02-14 05:34:23,208 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 80.95 MB 2025-02-14 05:34:23,208 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 16909.34 MB 2025-02-14 05:34:23,208 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17293.12 MB 2025-02-14 05:34:23,208 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 383.78 MB 2025-02-14 05:34:23,208 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18046.15 MB 2025-02-14 05:34:23,213 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 05:34:23,213 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 05:34:23,213 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 05:34:23,213 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:34:23,213 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14231.81 MB 2025-02-14 05:34:23,213 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14519.89 MB 2025-02-14 05:34:23,213 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 288.08 MB 2025-02-14 05:34:23,213 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17293.12 MB 2025-02-14 05:34:23,213 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17293.12 MB 2025-02-14 05:34:23,213 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:34:23,213 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14736.06 MB 2025-02-14 05:34:23,273 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 05:34:23,273 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 05:34:23,273 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 05:34:23,273 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:34:23,273 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14519.89 MB 2025-02-14 05:34:23,273 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14869.83 MB 2025-02-14 05:34:23,273 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 349.93 MB 2025-02-14 05:34:23,273 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17293.12 MB 2025-02-14 05:34:23,273 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17293.12 MB 2025-02-14 05:34:23,273 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:34:23,273 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15707.28 MB 2025-02-14 05:34:23,274 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 05:34:23,274 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 05:34:23,274 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 05:34:23,274 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:34:23,274 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14231.81 MB 2025-02-14 05:34:23,274 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14869.83 MB 2025-02-14 05:34:23,274 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 638.02 MB 2025-02-14 05:34:23,274 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17293.12 MB 2025-02-14 05:34:23,274 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17293.12 MB 2025-02-14 05:34:23,274 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:34:23,274 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15707.28 MB 2025-02-14 05:34:23,306 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 05:34:23,306 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 05:34:23,306 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 05:34:23,306 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:34:23,306 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15207.63 MB 2025-02-14 05:34:23,306 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15354.58 MB 2025-02-14 05:34:23,306 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 146.95 MB 2025-02-14 05:34:23,306 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17293.12 MB 2025-02-14 05:34:23,306 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17383.29 MB 2025-02-14 05:34:23,306 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 90.18 MB 2025-02-14 05:34:23,306 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15462.52 MB 2025-02-14 05:34:23,311 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 05:34:23,311 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 05:34:23,311 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 05:34:23,311 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:34:23,311 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15447.54 MB 2025-02-14 05:34:23,311 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15594.84 MB 2025-02-14 05:34:23,311 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 147.30 MB 2025-02-14 05:34:23,311 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17383.29 MB 2025-02-14 05:34:23,311 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17383.29 MB 2025-02-14 05:34:23,311 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:34:23,311 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15594.84 MB 2025-02-14 05:34:23,312 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 05:34:23,312 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 05:34:23,312 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.67 seconds 2025-02-14 05:34:23,312 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:34:23,312 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13473.90 MB 2025-02-14 05:34:23,312 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15726.68 MB 2025-02-14 05:34:23,312 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2252.79 MB 2025-02-14 05:34:23,312 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57440.99 MB 2025-02-14 05:34:23,312 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17383.29 MB 2025-02-14 05:34:23,312 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -40057.70 MB 2025-02-14 05:34:23,312 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15726.68 MB 2025-02-14 05:34:23,482 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 05:34:23,482 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 05:34:23,482 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 05:34:23,482 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:34:23,482 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15726.68 MB 2025-02-14 05:34:23,482 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15806.07 MB 2025-02-14 05:34:23,482 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 79.38 MB 2025-02-14 05:34:23,482 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17383.29 MB 2025-02-14 05:34:23,482 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18176.02 MB 2025-02-14 05:34:23,482 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 792.72 MB 2025-02-14 05:34:23,482 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17637.33 MB 2025-02-14 05:34:23,494 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 5347, cut from 5349 2025-02-14 05:34:23,494 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 05:34:23,499 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 05:34:23,499 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 05:34:23,499 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:34:23,499 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:34:23,499 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15806.07 MB 2025-02-14 05:34:23,499 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21339.13 MB 2025-02-14 05:34:23,499 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5533.07 MB 2025-02-14 05:34:23,499 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18176.02 MB 2025-02-14 05:34:23,499 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25054.67 MB 2025-02-14 05:34:23,499 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6878.66 MB 2025-02-14 05:34:23,499 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21339.13 MB 2025-02-14 05:34:23,609 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 5139] 2025-02-14 05:34:23,611 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:34:23,611 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 05:34:23,612 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:34:23,612 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 05:34:23,617 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 05:34:23,618 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:34:23,618 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 05:34:23,618 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 05:34:32,912 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:34:32,913 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 05:34:32,917 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 05:34:32,921 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:34:32,921 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1267, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 05:34:32,922 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:34:32,922 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1267, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 05:34:52,533 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 05:34:52,534 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 05:34:52,534 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.61 seconds 2025-02-14 05:34:52,534 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:34:52,534 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21797.37 MB 2025-02-14 05:34:52,534 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26281.21 MB 2025-02-14 05:34:52,534 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4483.84 MB 2025-02-14 05:34:52,534 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30557.60 MB 2025-02-14 05:34:52,534 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32398.90 MB 2025-02-14 05:34:52,534 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1841.30 MB 2025-02-14 05:34:52,534 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35119.11 MB 2025-02-14 05:34:52,611 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 05:34:52,611 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 05:34:52,611 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 05:34:52,611 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:34:52,611 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26281.21 MB 2025-02-14 05:34:52,611 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22364.59 MB 2025-02-14 05:34:52,611 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3916.62 MB 2025-02-14 05:34:52,611 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32398.90 MB 2025-02-14 05:34:52,611 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47232.06 MB 2025-02-14 05:34:52,611 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 14833.16 MB 2025-02-14 05:34:52,611 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39653.41 MB 2025-02-14 05:34:54,541 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 05:34:54,541 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 05:34:54,541 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 05:34:54,541 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:34:54,541 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22364.59 MB 2025-02-14 05:34:54,541 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22895.43 MB 2025-02-14 05:34:54,541 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 05:34:54,541 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47232.06 MB 2025-02-14 05:34:54,541 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26577.21 MB 2025-02-14 05:34:54,541 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20654.85 MB 2025-02-14 05:34:54,541 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26875.80 MB 2025-02-14 05:34:54,555 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 05:34:54,555 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 05:34:54,555 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 05:34:54,555 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:34:54,555 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22895.43 MB 2025-02-14 05:34:54,555 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24784.96 MB 2025-02-14 05:34:54,555 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 05:34:54,555 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26577.21 MB 2025-02-14 05:34:54,555 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29408.36 MB 2025-02-14 05:34:54,555 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-14 05:34:54,555 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26202.39 MB 2025-02-14 05:34:54,762 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 05:34:54,762 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 05:34:54,762 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 05:34:54,762 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:34:54,762 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24784.96 MB 2025-02-14 05:34:54,762 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27026.82 MB 2025-02-14 05:34:54,762 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 05:34:54,762 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29408.36 MB 2025-02-14 05:34:54,762 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35070.67 MB 2025-02-14 05:34:54,762 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 05:34:54,762 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32571.10 MB 2025-02-14 05:34:54,763 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 05:34:54,763 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 05:34:54,763 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 05:34:54,763 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:34:54,763 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22895.43 MB 2025-02-14 05:34:54,763 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27026.82 MB 2025-02-14 05:34:54,763 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 05:34:54,763 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26577.21 MB 2025-02-14 05:34:54,763 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35070.67 MB 2025-02-14 05:34:54,763 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8493.47 MB 2025-02-14 05:34:54,763 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32571.10 MB 2025-02-14 05:34:54,928 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 05:34:54,928 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 05:34:54,928 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 05:34:54,928 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:34:54,928 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28560.36 MB 2025-02-14 05:34:54,928 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29327.36 MB 2025-02-14 05:34:54,928 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 05:34:54,928 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35070.67 MB 2025-02-14 05:34:54,928 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35488.01 MB 2025-02-14 05:34:54,928 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 05:34:54,928 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30035.15 MB 2025-02-14 05:34:54,947 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 05:34:54,947 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 05:34:54,947 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:34:54,947 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:34:54,947 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29740.25 MB 2025-02-14 05:34:54,947 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29967.47 MB 2025-02-14 05:34:54,947 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.22 MB 2025-02-14 05:34:54,947 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35488.01 MB 2025-02-14 05:34:54,947 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35488.01 MB 2025-02-14 05:34:54,947 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:34:54,947 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30208.60 MB 2025-02-14 05:34:54,948 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 05:34:54,948 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 05:34:54,948 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.02 seconds 2025-02-14 05:34:54,948 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:34:54,948 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17383.04 MB 2025-02-14 05:34:54,948 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30168.37 MB 2025-02-14 05:34:54,948 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12785.33 MB 2025-02-14 05:34:54,948 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30557.60 MB 2025-02-14 05:34:54,948 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35488.01 MB 2025-02-14 05:34:54,948 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4930.40 MB 2025-02-14 05:34:54,948 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30208.60 MB 2025-02-14 05:34:55,218 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 05:34:55,219 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 05:34:55,219 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 05:34:55,219 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:34:55,219 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30168.37 MB 2025-02-14 05:34:55,219 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22384.76 MB 2025-02-14 05:34:55,219 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7783.61 MB 2025-02-14 05:34:55,219 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35488.01 MB 2025-02-14 05:34:55,219 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35488.01 MB 2025-02-14 05:34:55,219 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:34:55,219 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32677.89 MB 2025-02-14 05:34:55,237 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8155, cut from 8157 2025-02-14 05:34:55,237 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 05:34:55,243 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 05:34:55,243 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 05:34:55,243 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:34:55,243 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:34:55,243 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22384.76 MB 2025-02-14 05:34:55,243 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30816.22 MB 2025-02-14 05:34:55,243 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8431.46 MB 2025-02-14 05:34:55,243 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35488.01 MB 2025-02-14 05:34:55,243 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43872.42 MB 2025-02-14 05:34:55,243 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8384.41 MB 2025-02-14 05:34:55,243 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30816.22 MB 2025-02-14 05:34:55,404 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7947] 2025-02-14 05:34:55,405 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:34:55,405 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 05:34:55,406 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:34:55,406 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 05:34:55,411 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 05:34:55,412 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:34:55,412 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 05:34:55,412 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 05:35:01,990 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:35:01,991 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 05:35:01,996 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 05:35:01,999 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:35:01,999 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 176, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 05:35:02,000 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:35:02,000 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 176, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 05:35:04,805 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 05:35:04,805 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 05:35:04,805 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.80 seconds 2025-02-14 05:35:04,805 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:35:04,805 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14195.10 MB 2025-02-14 05:35:04,805 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14817.96 MB 2025-02-14 05:35:04,805 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 622.85 MB 2025-02-14 05:35:04,805 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52256.83 MB 2025-02-14 05:35:04,805 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19658.70 MB 2025-02-14 05:35:04,805 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32598.13 MB 2025-02-14 05:35:04,805 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23666.47 MB 2025-02-14 05:35:04,818 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 05:35:04,818 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 05:35:04,818 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 05:35:04,818 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:35:04,818 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14817.96 MB 2025-02-14 05:35:04,818 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15119.73 MB 2025-02-14 05:35:04,818 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 301.77 MB 2025-02-14 05:35:04,818 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19658.70 MB 2025-02-14 05:35:04,818 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19658.70 MB 2025-02-14 05:35:04,818 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:35:04,818 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17347.54 MB 2025-02-14 05:35:05,667 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 05:35:05,668 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 05:35:05,668 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.85 seconds 2025-02-14 05:35:05,668 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:35:05,668 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15119.73 MB 2025-02-14 05:35:05,668 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15353.30 MB 2025-02-14 05:35:05,668 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 233.57 MB 2025-02-14 05:35:05,668 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19658.70 MB 2025-02-14 05:35:05,668 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19658.70 MB 2025-02-14 05:35:05,668 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:35:05,668 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19291.20 MB 2025-02-14 05:35:05,676 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 05:35:05,676 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 05:35:05,676 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 05:35:05,676 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:35:05,676 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15353.23 MB 2025-02-14 05:35:05,676 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16184.43 MB 2025-02-14 05:35:05,676 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 831.19 MB 2025-02-14 05:35:05,676 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19658.70 MB 2025-02-14 05:35:05,676 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19658.70 MB 2025-02-14 05:35:05,676 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:35:05,676 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16808.10 MB 2025-02-14 05:35:05,773 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 05:35:05,773 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 05:35:05,773 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 05:35:05,773 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:35:05,773 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16184.43 MB 2025-02-14 05:35:05,773 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17170.88 MB 2025-02-14 05:35:05,773 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 986.45 MB 2025-02-14 05:35:05,773 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19658.70 MB 2025-02-14 05:35:05,773 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20904.41 MB 2025-02-14 05:35:05,773 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1245.71 MB 2025-02-14 05:35:05,773 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19610.33 MB 2025-02-14 05:35:05,774 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 05:35:05,774 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 05:35:05,774 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 05:35:05,774 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:35:05,774 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15353.23 MB 2025-02-14 05:35:05,774 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17170.88 MB 2025-02-14 05:35:05,774 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1817.65 MB 2025-02-14 05:35:05,774 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19658.70 MB 2025-02-14 05:35:05,774 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20904.41 MB 2025-02-14 05:35:05,774 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1245.71 MB 2025-02-14 05:35:05,774 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19610.33 MB 2025-02-14 05:35:05,849 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 05:35:05,849 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 05:35:05,849 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 05:35:05,849 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:35:05,849 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17845.64 MB 2025-02-14 05:35:05,849 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18183.12 MB 2025-02-14 05:35:05,849 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 337.48 MB 2025-02-14 05:35:05,849 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20904.41 MB 2025-02-14 05:35:05,849 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21086.86 MB 2025-02-14 05:35:05,849 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 182.45 MB 2025-02-14 05:35:05,849 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18500.80 MB 2025-02-14 05:35:05,859 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 05:35:05,859 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 05:35:05,859 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 05:35:05,859 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:35:05,859 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18364.80 MB 2025-02-14 05:35:05,859 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18593.70 MB 2025-02-14 05:35:05,859 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.91 MB 2025-02-14 05:35:05,859 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21086.86 MB 2025-02-14 05:35:05,859 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21086.86 MB 2025-02-14 05:35:05,859 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:35:05,859 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18629.79 MB 2025-02-14 05:35:05,860 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 05:35:05,860 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 05:35:05,860 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.86 seconds 2025-02-14 05:35:05,860 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:35:05,860 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13581.90 MB 2025-02-14 05:35:05,860 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18794.55 MB 2025-02-14 05:35:05,860 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5212.65 MB 2025-02-14 05:35:05,860 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52256.83 MB 2025-02-14 05:35:05,860 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21086.86 MB 2025-02-14 05:35:05,860 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31169.97 MB 2025-02-14 05:35:05,860 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18794.55 MB 2025-02-14 05:35:06,128 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 05:35:06,128 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 05:35:06,128 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 05:35:06,128 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:35:06,128 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18794.55 MB 2025-02-14 05:35:06,128 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17525.74 MB 2025-02-14 05:35:06,128 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1268.81 MB 2025-02-14 05:35:06,128 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21086.86 MB 2025-02-14 05:35:06,128 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21086.86 MB 2025-02-14 05:35:06,128 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:35:06,128 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19028.72 MB 2025-02-14 05:35:06,147 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8153, cut from 8155 2025-02-14 05:35:06,147 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2 ('] 2025-02-14 05:35:06,153 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 05:35:06,153 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 05:35:06,153 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:35:06,153 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:35:06,153 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17525.74 MB 2025-02-14 05:35:06,153 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25956.14 MB 2025-02-14 05:35:06,153 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8430.40 MB 2025-02-14 05:35:06,153 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21086.86 MB 2025-02-14 05:35:06,153 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31562.14 MB 2025-02-14 05:35:06,153 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10475.27 MB 2025-02-14 05:35:06,153 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25956.14 MB 2025-02-14 05:35:06,319 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7945] 2025-02-14 05:35:06,320 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:35:06,320 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 05:35:06,321 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:35:06,321 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 05:35:06,326 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 05:35:06,327 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:35:06,327 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 05:35:06,327 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2 ('] 2025-02-14 05:37:02,904 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:37:02,904 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 05:37:02,909 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 05:37:02,913 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:37:02,913 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 86, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 05:37:02,914 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:37:02,914 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 86, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 05:37:04,249 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 05:37:04,249 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 05:37:04,249 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.33 seconds 2025-02-14 05:37:04,249 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:37:04,249 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13567.97 MB 2025-02-14 05:37:04,249 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 13872.32 MB 2025-02-14 05:37:04,249 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 304.35 MB 2025-02-14 05:37:04,249 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39942.36 MB 2025-02-14 05:37:04,249 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17322.48 MB 2025-02-14 05:37:04,249 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22619.88 MB 2025-02-14 05:37:04,249 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22813.42 MB 2025-02-14 05:37:04,251 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 05:37:04,252 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 05:37:04,252 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 05:37:04,252 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:37:04,252 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13872.32 MB 2025-02-14 05:37:04,252 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14019.77 MB 2025-02-14 05:37:04,252 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 147.46 MB 2025-02-14 05:37:04,252 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17322.48 MB 2025-02-14 05:37:04,252 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17322.48 MB 2025-02-14 05:37:04,252 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:37:04,252 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14476.36 MB 2025-02-14 05:37:04,667 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 05:37:04,667 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 05:37:04,667 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.41 seconds 2025-02-14 05:37:04,667 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:37:04,667 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14019.77 MB 2025-02-14 05:37:04,667 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14133.90 MB 2025-02-14 05:37:04,667 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 114.13 MB 2025-02-14 05:37:04,667 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17322.48 MB 2025-02-14 05:37:04,667 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17322.48 MB 2025-02-14 05:37:04,667 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:37:04,667 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18106.31 MB 2025-02-14 05:37:04,672 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 05:37:04,672 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 05:37:04,672 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 05:37:04,672 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:37:04,672 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14133.84 MB 2025-02-14 05:37:04,672 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14539.99 MB 2025-02-14 05:37:04,672 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 406.15 MB 2025-02-14 05:37:04,672 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17322.48 MB 2025-02-14 05:37:04,673 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17322.48 MB 2025-02-14 05:37:04,673 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:37:04,673 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14844.74 MB 2025-02-14 05:37:04,759 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 05:37:04,759 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 05:37:04,759 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 05:37:04,759 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:37:04,759 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14539.99 MB 2025-02-14 05:37:04,759 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15033.31 MB 2025-02-14 05:37:04,759 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 493.32 MB 2025-02-14 05:37:04,759 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17322.48 MB 2025-02-14 05:37:04,759 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17322.48 MB 2025-02-14 05:37:04,759 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:37:04,759 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16214.01 MB 2025-02-14 05:37:04,760 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 05:37:04,760 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 05:37:04,760 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 05:37:04,760 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:37:04,760 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14133.84 MB 2025-02-14 05:37:04,760 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15033.31 MB 2025-02-14 05:37:04,760 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 899.47 MB 2025-02-14 05:37:04,760 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17322.48 MB 2025-02-14 05:37:04,760 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17322.48 MB 2025-02-14 05:37:04,760 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:37:04,760 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16214.01 MB 2025-02-14 05:37:04,804 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 05:37:04,804 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 05:37:04,804 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-14 05:37:04,804 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:37:04,804 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15509.56 MB 2025-02-14 05:37:04,804 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15717.72 MB 2025-02-14 05:37:04,804 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 208.16 MB 2025-02-14 05:37:04,804 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17322.48 MB 2025-02-14 05:37:04,804 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17454.60 MB 2025-02-14 05:37:04,804 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 132.12 MB 2025-02-14 05:37:04,804 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15869.90 MB 2025-02-14 05:37:04,811 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 05:37:04,811 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 05:37:04,811 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 05:37:04,811 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:37:04,811 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15849.36 MB 2025-02-14 05:37:04,811 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16055.22 MB 2025-02-14 05:37:04,811 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.85 MB 2025-02-14 05:37:04,811 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17454.60 MB 2025-02-14 05:37:04,811 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17454.60 MB 2025-02-14 05:37:04,811 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:37:04,811 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16055.22 MB 2025-02-14 05:37:04,813 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 05:37:04,813 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 05:37:04,813 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.90 seconds 2025-02-14 05:37:04,813 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:37:04,813 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13268.34 MB 2025-02-14 05:37:04,813 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16238.70 MB 2025-02-14 05:37:04,813 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2970.37 MB 2025-02-14 05:37:04,813 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39942.36 MB 2025-02-14 05:37:04,813 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17454.60 MB 2025-02-14 05:37:04,813 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22487.76 MB 2025-02-14 05:37:04,813 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16238.70 MB 2025-02-14 05:37:05,056 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 05:37:05,057 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 05:37:05,057 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.24 seconds 2025-02-14 05:37:05,057 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:37:05,057 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13769.01 MB 2025-02-14 05:37:05,057 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16519.46 MB 2025-02-14 05:37:05,057 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2750.45 MB 2025-02-14 05:37:05,057 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17454.60 MB 2025-02-14 05:37:05,057 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17949.52 MB 2025-02-14 05:37:05,057 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 494.93 MB 2025-02-14 05:37:05,057 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16794.47 MB 2025-02-14 05:37:05,073 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 7447, cut from 7449 2025-02-14 05:37:05,073 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 video rate for this video is 2 ('] 2025-02-14 05:37:05,079 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 05:37:05,079 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 05:37:05,079 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:37:05,079 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:37:05,079 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16519.46 MB 2025-02-14 05:37:05,079 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24220.28 MB 2025-02-14 05:37:05,079 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7700.82 MB 2025-02-14 05:37:05,079 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17949.52 MB 2025-02-14 05:37:05,079 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27520.93 MB 2025-02-14 05:37:05,079 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9571.40 MB 2025-02-14 05:37:05,079 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24220.28 MB 2025-02-14 05:37:05,230 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7239] 2025-02-14 05:37:05,231 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:37:05,231 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 05:37:05,232 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:37:05,232 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 05:37:05,237 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 05:37:05,238 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:37:05,238 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 05:37:05,238 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 video rate for this video is 2 ('] 2025-02-14 05:39:15,361 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:39:15,362 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 05:39:15,370 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 05:39:15,377 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:39:15,377 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2583, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 05:39:15,379 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:39:15,379 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2583, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 05:39:54,972 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 05:39:54,972 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 05:39:54,972 - resource_logging.py:150 - __exit__ - DEBUG - Time: 39.58 seconds 2025-02-14 05:39:54,972 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:39:54,972 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30968.17 MB 2025-02-14 05:39:54,972 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40109.65 MB 2025-02-14 05:39:54,972 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 9141.49 MB 2025-02-14 05:39:54,972 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57006.88 MB 2025-02-14 05:39:54,972 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44050.68 MB 2025-02-14 05:39:54,972 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12956.21 MB 2025-02-14 05:39:54,972 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49250.75 MB 2025-02-14 05:39:55,141 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 05:39:55,141 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 05:39:55,141 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 05:39:55,141 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:39:55,141 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40109.65 MB 2025-02-14 05:39:55,141 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29206.77 MB 2025-02-14 05:39:55,141 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10902.89 MB 2025-02-14 05:39:55,141 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44050.68 MB 2025-02-14 05:39:55,141 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 78479.62 MB 2025-02-14 05:39:55,141 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 34428.94 MB 2025-02-14 05:39:55,141 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 66469.30 MB 2025-02-14 05:39:57,109 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 05:39:57,109 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 05:39:57,109 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.97 seconds 2025-02-14 05:39:57,109 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:39:57,109 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29206.77 MB 2025-02-14 05:39:57,109 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29737.61 MB 2025-02-14 05:39:57,109 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 05:39:57,109 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 78479.62 MB 2025-02-14 05:39:57,109 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31752.98 MB 2025-02-14 05:39:57,109 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -46726.64 MB 2025-02-14 05:39:57,109 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33717.98 MB 2025-02-14 05:39:57,123 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 05:39:57,123 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 05:39:57,123 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 05:39:57,123 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:39:57,123 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29737.61 MB 2025-02-14 05:39:57,123 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31627.14 MB 2025-02-14 05:39:57,123 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 05:39:57,123 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31752.98 MB 2025-02-14 05:39:57,123 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35055.99 MB 2025-02-14 05:39:57,123 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 05:39:57,123 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33044.57 MB 2025-02-14 05:39:57,331 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 05:39:57,331 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 05:39:57,331 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 05:39:57,331 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:39:57,331 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31627.14 MB 2025-02-14 05:39:57,331 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33869.00 MB 2025-02-14 05:39:57,331 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 05:39:57,331 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35055.99 MB 2025-02-14 05:39:57,331 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41662.02 MB 2025-02-14 05:39:57,331 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 05:39:57,331 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39413.28 MB 2025-02-14 05:39:57,332 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 05:39:57,332 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 05:39:57,332 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 05:39:57,332 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:39:57,332 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29737.61 MB 2025-02-14 05:39:57,332 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33869.00 MB 2025-02-14 05:39:57,332 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 05:39:57,332 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31752.98 MB 2025-02-14 05:39:57,332 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41662.02 MB 2025-02-14 05:39:57,332 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9909.04 MB 2025-02-14 05:39:57,332 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39413.28 MB 2025-02-14 05:39:57,497 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 05:39:57,497 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 05:39:57,497 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 05:39:57,497 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:39:57,497 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35402.54 MB 2025-02-14 05:39:57,497 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36169.54 MB 2025-02-14 05:39:57,497 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 05:39:57,497 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41662.02 MB 2025-02-14 05:39:57,497 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42079.35 MB 2025-02-14 05:39:57,497 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 05:39:57,497 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36877.33 MB 2025-02-14 05:39:57,515 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 05:39:57,515 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 05:39:57,515 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:39:57,516 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:39:57,516 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36582.43 MB 2025-02-14 05:39:57,516 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36809.79 MB 2025-02-14 05:39:57,516 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.36 MB 2025-02-14 05:39:57,516 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42079.35 MB 2025-02-14 05:39:57,516 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42079.35 MB 2025-02-14 05:39:57,516 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:39:57,516 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37029.84 MB 2025-02-14 05:39:57,517 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 05:39:57,517 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 05:39:57,517 - resource_logging.py:150 - __exit__ - DEBUG - Time: 42.13 seconds 2025-02-14 05:39:57,517 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:39:57,517 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21968.44 MB 2025-02-14 05:39:57,517 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37010.55 MB 2025-02-14 05:39:57,517 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 15042.11 MB 2025-02-14 05:39:57,517 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48005.91 MB 2025-02-14 05:39:57,517 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42079.35 MB 2025-02-14 05:39:57,517 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5926.55 MB 2025-02-14 05:39:57,517 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37029.84 MB 2025-02-14 05:39:57,787 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 05:39:57,787 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 05:39:57,788 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 05:39:57,788 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:39:57,788 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37010.55 MB 2025-02-14 05:39:57,788 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26967.87 MB 2025-02-14 05:39:57,788 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10042.67 MB 2025-02-14 05:39:57,788 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42079.35 MB 2025-02-14 05:39:57,788 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42079.35 MB 2025-02-14 05:39:57,788 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:39:57,788 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39518.22 MB 2025-02-14 05:39:57,805 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8149, cut from 8151 2025-02-14 05:39:57,806 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2 ('] 2025-02-14 05:39:57,811 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 05:39:57,811 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 05:39:57,811 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:39:57,812 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:39:57,812 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26967.87 MB 2025-02-14 05:39:57,812 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35393.34 MB 2025-02-14 05:39:57,812 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8425.47 MB 2025-02-14 05:39:57,812 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42079.35 MB 2025-02-14 05:39:57,812 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46267.37 MB 2025-02-14 05:39:57,812 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4188.01 MB 2025-02-14 05:39:57,812 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35393.34 MB 2025-02-14 05:39:57,973 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7941] 2025-02-14 05:39:57,974 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:39:57,974 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 05:39:57,975 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:39:57,975 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 05:39:57,980 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 05:39:57,981 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:39:57,981 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 05:39:57,981 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2 ('] 2025-02-14 05:40:43,404 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:40:43,404 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 05:40:43,409 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 05:40:43,413 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:40:43,413 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2959, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 05:40:43,414 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:40:43,414 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2959, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 05:41:29,457 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 05:41:29,457 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 05:41:29,457 - resource_logging.py:150 - __exit__ - DEBUG - Time: 46.03 seconds 2025-02-14 05:41:29,457 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:41:29,457 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33588.93 MB 2025-02-14 05:41:29,457 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44060.67 MB 2025-02-14 05:41:29,457 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 10471.74 MB 2025-02-14 05:41:29,457 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 75266.79 MB 2025-02-14 05:41:29,457 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48003.81 MB 2025-02-14 05:41:29,457 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27262.98 MB 2025-02-14 05:41:29,457 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54532.40 MB 2025-02-14 05:41:29,804 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 05:41:29,804 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 05:41:29,804 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.35 seconds 2025-02-14 05:41:29,804 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:41:29,804 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44060.67 MB 2025-02-14 05:41:29,804 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31161.49 MB 2025-02-14 05:41:29,804 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -12899.18 MB 2025-02-14 05:41:29,804 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48003.81 MB 2025-02-14 05:41:29,804 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 88497.72 MB 2025-02-14 05:41:29,804 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 40493.91 MB 2025-02-14 05:41:29,804 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 74915.17 MB 2025-02-14 05:41:31,786 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 05:41:31,786 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 05:41:31,786 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.98 seconds 2025-02-14 05:41:31,786 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:41:31,786 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31161.49 MB 2025-02-14 05:41:31,786 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31692.33 MB 2025-02-14 05:41:31,786 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 05:41:31,786 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 88497.72 MB 2025-02-14 05:41:31,786 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33709.62 MB 2025-02-14 05:41:31,786 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -54788.10 MB 2025-02-14 05:41:31,786 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35672.70 MB 2025-02-14 05:41:31,800 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 05:41:31,800 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 05:41:31,800 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 05:41:31,800 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:41:31,800 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31692.33 MB 2025-02-14 05:41:31,800 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33581.86 MB 2025-02-14 05:41:31,800 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 05:41:31,800 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33709.62 MB 2025-02-14 05:41:31,800 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37012.64 MB 2025-02-14 05:41:31,800 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 05:41:31,800 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34999.29 MB 2025-02-14 05:41:32,011 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 05:41:32,012 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 05:41:32,012 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 05:41:32,012 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:41:32,012 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33581.86 MB 2025-02-14 05:41:32,012 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35823.72 MB 2025-02-14 05:41:32,012 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 05:41:32,012 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37012.64 MB 2025-02-14 05:41:32,012 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43618.66 MB 2025-02-14 05:41:32,012 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 05:41:32,012 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41368.00 MB 2025-02-14 05:41:32,012 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 05:41:32,012 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 05:41:32,012 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 05:41:32,012 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:41:32,012 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31692.33 MB 2025-02-14 05:41:32,012 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35823.72 MB 2025-02-14 05:41:32,012 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 05:41:32,013 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33709.62 MB 2025-02-14 05:41:32,013 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43618.66 MB 2025-02-14 05:41:32,013 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9909.04 MB 2025-02-14 05:41:32,013 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41368.00 MB 2025-02-14 05:41:32,218 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 05:41:32,219 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 05:41:32,219 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 05:41:32,219 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:41:32,219 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37357.26 MB 2025-02-14 05:41:32,219 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38124.26 MB 2025-02-14 05:41:32,219 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 05:41:32,219 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43618.66 MB 2025-02-14 05:41:32,219 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44036.00 MB 2025-02-14 05:41:32,219 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 05:41:32,219 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38832.05 MB 2025-02-14 05:41:32,238 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 05:41:32,238 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 05:41:32,238 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:41:32,238 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:41:32,238 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38537.15 MB 2025-02-14 05:41:32,238 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38764.86 MB 2025-02-14 05:41:32,238 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.71 MB 2025-02-14 05:41:32,238 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44036.00 MB 2025-02-14 05:41:32,238 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44036.00 MB 2025-02-14 05:41:32,238 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:41:32,238 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38977.68 MB 2025-02-14 05:41:32,240 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 05:41:32,240 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 05:41:32,240 - resource_logging.py:150 - __exit__ - DEBUG - Time: 48.82 seconds 2025-02-14 05:41:32,240 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:41:32,240 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23278.82 MB 2025-02-14 05:41:32,240 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38965.00 MB 2025-02-14 05:41:32,240 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 15686.18 MB 2025-02-14 05:41:32,240 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64955.09 MB 2025-02-14 05:41:32,240 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44036.00 MB 2025-02-14 05:41:32,240 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20919.09 MB 2025-02-14 05:41:32,240 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38977.68 MB 2025-02-14 05:41:32,508 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 05:41:32,508 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 05:41:32,508 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 05:41:32,508 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:41:32,508 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25268.19 MB 2025-02-14 05:41:32,508 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28268.22 MB 2025-02-14 05:41:32,508 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3000.03 MB 2025-02-14 05:41:32,508 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44036.00 MB 2025-02-14 05:41:32,508 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44036.00 MB 2025-02-14 05:41:32,508 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:41:32,508 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28568.19 MB 2025-02-14 05:41:32,526 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8124, cut from 8126 2025-02-14 05:41:32,526 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 1 ('] 2025-02-14 05:41:32,532 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 05:41:32,532 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 05:41:32,532 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:41:32,532 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:41:32,532 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28268.22 MB 2025-02-14 05:41:32,532 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36668.13 MB 2025-02-14 05:41:32,532 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8399.91 MB 2025-02-14 05:41:32,532 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44036.00 MB 2025-02-14 05:41:32,532 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48211.43 MB 2025-02-14 05:41:32,532 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4175.43 MB 2025-02-14 05:41:32,532 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36668.13 MB 2025-02-14 05:41:32,694 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7916] 2025-02-14 05:41:32,695 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:41:32,695 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 05:41:32,696 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:41:32,696 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 05:41:32,701 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 05:41:32,702 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:41:32,702 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 05:41:32,702 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 1 ('] 2025-02-14 05:42:23,771 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:42:23,771 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 05:42:23,776 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 05:42:23,780 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:42:23,780 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1069, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 05:42:23,781 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:42:23,781 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1069, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 05:42:40,348 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 05:42:40,348 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 05:42:40,348 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.56 seconds 2025-02-14 05:42:40,348 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:42:40,348 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20417.67 MB 2025-02-14 05:42:40,348 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24200.93 MB 2025-02-14 05:42:40,348 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3783.26 MB 2025-02-14 05:42:40,349 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56562.29 MB 2025-02-14 05:42:40,349 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29041.36 MB 2025-02-14 05:42:40,349 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27520.93 MB 2025-02-14 05:42:40,349 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33060.74 MB 2025-02-14 05:42:40,411 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 05:42:40,411 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 05:42:40,411 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 05:42:40,411 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:42:40,411 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24200.93 MB 2025-02-14 05:42:40,411 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21335.25 MB 2025-02-14 05:42:40,411 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2865.68 MB 2025-02-14 05:42:40,411 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29041.36 MB 2025-02-14 05:42:40,411 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39283.85 MB 2025-02-14 05:42:40,411 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10242.49 MB 2025-02-14 05:42:40,411 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34476.80 MB 2025-02-14 05:42:42,334 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 05:42:42,334 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 05:42:42,334 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 05:42:42,334 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:42:42,334 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21335.25 MB 2025-02-14 05:42:42,334 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21866.09 MB 2025-02-14 05:42:42,334 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 05:42:42,334 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39283.85 MB 2025-02-14 05:42:42,334 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26673.68 MB 2025-02-14 05:42:42,334 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12610.17 MB 2025-02-14 05:42:42,334 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25845.42 MB 2025-02-14 05:42:42,347 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 05:42:42,348 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 05:42:42,348 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 05:42:42,348 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:42:42,348 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21866.09 MB 2025-02-14 05:42:42,348 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23755.62 MB 2025-02-14 05:42:42,348 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 05:42:42,348 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26673.68 MB 2025-02-14 05:42:42,348 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28561.11 MB 2025-02-14 05:42:42,348 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 05:42:42,348 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25173.05 MB 2025-02-14 05:42:42,557 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 05:42:42,557 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 05:42:42,557 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 05:42:42,557 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:42:42,557 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23755.62 MB 2025-02-14 05:42:42,557 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25997.48 MB 2025-02-14 05:42:42,557 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 05:42:42,557 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28561.11 MB 2025-02-14 05:42:42,557 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34223.42 MB 2025-02-14 05:42:42,557 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 05:42:42,557 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31541.76 MB 2025-02-14 05:42:42,558 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 05:42:42,558 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 05:42:42,558 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 05:42:42,558 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:42:42,558 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21866.09 MB 2025-02-14 05:42:42,558 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25997.48 MB 2025-02-14 05:42:42,558 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 05:42:42,558 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26673.68 MB 2025-02-14 05:42:42,558 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34223.42 MB 2025-02-14 05:42:42,558 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-14 05:42:42,558 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31541.76 MB 2025-02-14 05:42:42,723 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 05:42:42,723 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 05:42:42,723 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 05:42:42,723 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:42:42,723 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27531.02 MB 2025-02-14 05:42:42,723 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28298.02 MB 2025-02-14 05:42:42,723 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 05:42:42,723 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34223.42 MB 2025-02-14 05:42:42,723 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34640.76 MB 2025-02-14 05:42:42,723 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 05:42:42,724 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29005.81 MB 2025-02-14 05:42:42,742 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 05:42:42,742 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 05:42:42,742 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:42:42,742 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:42:42,742 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28710.91 MB 2025-02-14 05:42:42,742 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28939.54 MB 2025-02-14 05:42:42,742 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.63 MB 2025-02-14 05:42:42,742 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34640.76 MB 2025-02-14 05:42:42,742 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34640.76 MB 2025-02-14 05:42:42,742 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:42:42,742 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29158.77 MB 2025-02-14 05:42:42,743 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 05:42:42,743 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 05:42:42,743 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.96 seconds 2025-02-14 05:42:42,743 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:42:42,743 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16693.19 MB 2025-02-14 05:42:42,743 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29140.61 MB 2025-02-14 05:42:42,743 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12447.43 MB 2025-02-14 05:42:42,743 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56562.29 MB 2025-02-14 05:42:42,743 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34640.76 MB 2025-02-14 05:42:42,743 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21921.53 MB 2025-02-14 05:42:42,743 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29158.77 MB 2025-02-14 05:42:43,012 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 05:42:43,012 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 05:42:43,012 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 05:42:43,012 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:42:43,012 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29140.61 MB 2025-02-14 05:42:43,012 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21697.58 MB 2025-02-14 05:42:43,012 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7443.04 MB 2025-02-14 05:42:43,012 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34640.76 MB 2025-02-14 05:42:43,012 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34640.76 MB 2025-02-14 05:42:43,012 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:42:43,012 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31652.28 MB 2025-02-14 05:42:43,030 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 05:42:43,030 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 05:42:43,036 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 05:42:43,036 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 05:42:43,036 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:42:43,036 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:42:43,036 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21697.58 MB 2025-02-14 05:42:43,036 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30136.60 MB 2025-02-14 05:42:43,036 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 05:42:43,036 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34640.76 MB 2025-02-14 05:42:43,036 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43031.46 MB 2025-02-14 05:42:43,036 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 05:42:43,036 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30136.60 MB 2025-02-14 05:42:43,199 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 05:42:43,200 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:42:43,200 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 05:42:43,201 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:42:43,201 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 05:42:43,206 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 05:42:43,207 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:42:43,207 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 05:42:43,207 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 05:42:55,436 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:42:55,436 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 05:42:55,441 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 05:42:55,444 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:42:55,444 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1017, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 05:42:55,445 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:42:55,445 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1017, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 05:43:11,353 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 05:43:11,353 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 05:43:11,353 - resource_logging.py:150 - __exit__ - DEBUG - Time: 15.90 seconds 2025-02-14 05:43:11,353 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:43:11,353 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20055.33 MB 2025-02-14 05:43:11,353 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23654.43 MB 2025-02-14 05:43:11,353 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3599.11 MB 2025-02-14 05:43:11,353 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55616.47 MB 2025-02-14 05:43:11,353 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28863.10 MB 2025-02-14 05:43:11,353 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26753.37 MB 2025-02-14 05:43:11,353 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32471.90 MB 2025-02-14 05:43:11,413 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 05:43:11,413 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 05:43:11,413 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 05:43:11,413 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:43:11,413 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23654.43 MB 2025-02-14 05:43:11,413 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21064.92 MB 2025-02-14 05:43:11,413 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2589.51 MB 2025-02-14 05:43:11,413 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28863.10 MB 2025-02-14 05:43:11,413 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39183.19 MB 2025-02-14 05:43:11,413 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10320.08 MB 2025-02-14 05:43:11,413 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34169.96 MB 2025-02-14 05:43:13,335 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 05:43:13,335 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 05:43:13,335 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 05:43:13,335 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:43:13,335 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21064.92 MB 2025-02-14 05:43:13,335 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21595.76 MB 2025-02-14 05:43:13,335 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 05:43:13,335 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39183.19 MB 2025-02-14 05:43:13,335 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26677.87 MB 2025-02-14 05:43:13,335 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12505.32 MB 2025-02-14 05:43:13,335 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25575.09 MB 2025-02-14 05:43:13,348 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 05:43:13,348 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 05:43:13,348 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 05:43:13,348 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:43:13,348 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21595.76 MB 2025-02-14 05:43:13,348 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23485.29 MB 2025-02-14 05:43:13,348 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 05:43:13,348 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26677.87 MB 2025-02-14 05:43:13,348 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28565.31 MB 2025-02-14 05:43:13,348 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 05:43:13,349 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24902.72 MB 2025-02-14 05:43:13,558 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 05:43:13,558 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 05:43:13,558 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 05:43:13,558 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:43:13,558 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23485.29 MB 2025-02-14 05:43:13,558 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25727.15 MB 2025-02-14 05:43:13,558 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 05:43:13,558 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28565.31 MB 2025-02-14 05:43:13,558 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34227.62 MB 2025-02-14 05:43:13,558 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 05:43:13,558 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31271.43 MB 2025-02-14 05:43:13,559 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 05:43:13,559 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 05:43:13,559 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 05:43:13,559 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:43:13,559 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21595.76 MB 2025-02-14 05:43:13,559 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25727.15 MB 2025-02-14 05:43:13,559 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 05:43:13,559 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26677.87 MB 2025-02-14 05:43:13,559 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34227.62 MB 2025-02-14 05:43:13,559 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-14 05:43:13,559 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31271.43 MB 2025-02-14 05:43:13,725 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 05:43:13,725 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 05:43:13,725 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 05:43:13,725 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:43:13,725 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27260.69 MB 2025-02-14 05:43:13,725 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28027.69 MB 2025-02-14 05:43:13,725 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 05:43:13,725 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34227.62 MB 2025-02-14 05:43:13,725 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34640.76 MB 2025-02-14 05:43:13,725 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 05:43:13,725 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28735.48 MB 2025-02-14 05:43:13,743 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 05:43:13,744 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 05:43:13,744 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:43:13,744 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:43:13,744 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28440.58 MB 2025-02-14 05:43:13,744 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28666.59 MB 2025-02-14 05:43:13,744 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.01 MB 2025-02-14 05:43:13,744 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34640.76 MB 2025-02-14 05:43:13,744 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34640.76 MB 2025-02-14 05:43:13,744 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:43:13,744 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28843.34 MB 2025-02-14 05:43:13,745 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 05:43:13,745 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 05:43:13,745 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.30 seconds 2025-02-14 05:43:13,745 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:43:13,745 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16512.02 MB 2025-02-14 05:43:13,745 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28867.67 MB 2025-02-14 05:43:13,745 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12355.65 MB 2025-02-14 05:43:13,745 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55616.47 MB 2025-02-14 05:43:13,745 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34640.76 MB 2025-02-14 05:43:13,745 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20975.71 MB 2025-02-14 05:43:13,745 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28867.67 MB 2025-02-14 05:43:14,016 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 05:43:14,016 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 05:43:14,016 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 05:43:14,016 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:43:14,016 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28867.67 MB 2025-02-14 05:43:14,016 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21516.41 MB 2025-02-14 05:43:14,016 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7351.26 MB 2025-02-14 05:43:14,016 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34640.76 MB 2025-02-14 05:43:14,016 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34640.76 MB 2025-02-14 05:43:14,016 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:43:14,016 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31379.33 MB 2025-02-14 05:43:14,034 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 05:43:14,035 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 05:43:14,041 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 05:43:14,041 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 05:43:14,041 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:43:14,041 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:43:14,041 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21516.41 MB 2025-02-14 05:43:14,041 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29955.43 MB 2025-02-14 05:43:14,041 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 05:43:14,041 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34640.76 MB 2025-02-14 05:43:14,041 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45130.71 MB 2025-02-14 05:43:14,041 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 05:43:14,041 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29955.43 MB 2025-02-14 05:43:14,203 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 05:43:14,205 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:43:14,205 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 05:43:14,205 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:43:14,206 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 05:43:14,210 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 05:43:14,211 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:43:14,211 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 05:43:14,211 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 05:44:10,394 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:44:10,395 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 05:44:10,400 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 05:44:10,403 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:44:10,403 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 229, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 05:44:10,404 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:44:10,404 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 229, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 05:44:13,954 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 05:44:13,954 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 05:44:13,954 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.55 seconds 2025-02-14 05:44:13,954 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:44:13,954 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14564.42 MB 2025-02-14 05:44:13,954 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15374.83 MB 2025-02-14 05:44:13,954 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 810.42 MB 2025-02-14 05:44:13,954 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57715.72 MB 2025-02-14 05:44:13,954 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 16909.34 MB 2025-02-14 05:44:13,954 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -40806.38 MB 2025-02-14 05:44:13,954 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24263.09 MB 2025-02-14 05:44:13,970 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 05:44:13,970 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 05:44:13,970 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 05:44:13,970 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:44:13,970 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15374.83 MB 2025-02-14 05:44:13,970 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15619.93 MB 2025-02-14 05:44:13,970 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 245.10 MB 2025-02-14 05:44:13,970 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 16909.34 MB 2025-02-14 05:44:13,970 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19224.59 MB 2025-02-14 05:44:13,970 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2315.26 MB 2025-02-14 05:44:13,970 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18315.15 MB 2025-02-14 05:44:14,970 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 05:44:14,971 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 05:44:14,971 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.00 seconds 2025-02-14 05:44:14,971 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:44:14,971 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15619.93 MB 2025-02-14 05:44:14,971 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15895.97 MB 2025-02-14 05:44:14,971 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 276.04 MB 2025-02-14 05:44:14,971 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19224.59 MB 2025-02-14 05:44:14,971 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18048.09 MB 2025-02-14 05:44:14,971 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1176.50 MB 2025-02-14 05:44:14,971 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19877.38 MB 2025-02-14 05:44:14,979 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 05:44:14,979 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 05:44:14,979 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 05:44:14,979 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:44:14,979 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15895.97 MB 2025-02-14 05:44:14,979 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16878.29 MB 2025-02-14 05:44:14,979 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 982.32 MB 2025-02-14 05:44:14,979 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18048.09 MB 2025-02-14 05:44:14,979 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18538.82 MB 2025-02-14 05:44:14,979 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 490.73 MB 2025-02-14 05:44:14,979 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17615.35 MB 2025-02-14 05:44:15,090 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 05:44:15,090 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 05:44:15,090 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 05:44:15,090 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:44:15,090 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16878.29 MB 2025-02-14 05:44:15,090 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18044.08 MB 2025-02-14 05:44:15,090 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1165.80 MB 2025-02-14 05:44:15,090 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18538.82 MB 2025-02-14 05:44:15,090 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21973.96 MB 2025-02-14 05:44:15,090 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3435.13 MB 2025-02-14 05:44:15,090 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20927.60 MB 2025-02-14 05:44:15,091 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 05:44:15,091 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 05:44:15,091 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 05:44:15,091 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:44:15,091 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15895.97 MB 2025-02-14 05:44:15,091 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18044.08 MB 2025-02-14 05:44:15,091 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2148.11 MB 2025-02-14 05:44:15,091 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18048.09 MB 2025-02-14 05:44:15,091 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21973.96 MB 2025-02-14 05:44:15,091 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3925.87 MB 2025-02-14 05:44:15,091 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20927.60 MB 2025-02-14 05:44:15,174 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 05:44:15,174 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 05:44:15,174 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 05:44:15,174 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:44:15,174 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18842.05 MB 2025-02-14 05:44:15,174 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19240.89 MB 2025-02-14 05:44:15,174 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 398.84 MB 2025-02-14 05:44:15,174 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21973.96 MB 2025-02-14 05:44:15,174 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22187.87 MB 2025-02-14 05:44:15,174 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 213.91 MB 2025-02-14 05:44:15,174 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19611.24 MB 2025-02-14 05:44:15,185 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 05:44:15,185 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 05:44:15,185 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 05:44:15,185 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:44:15,185 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19455.60 MB 2025-02-14 05:44:15,185 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19684.53 MB 2025-02-14 05:44:15,185 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.94 MB 2025-02-14 05:44:15,185 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22187.87 MB 2025-02-14 05:44:15,185 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22187.87 MB 2025-02-14 05:44:15,185 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:44:15,185 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19726.49 MB 2025-02-14 05:44:15,186 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 05:44:15,187 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 05:44:15,187 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.78 seconds 2025-02-14 05:44:15,187 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:44:15,187 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13766.56 MB 2025-02-14 05:44:15,187 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19885.41 MB 2025-02-14 05:44:15,187 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6118.85 MB 2025-02-14 05:44:15,187 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57715.72 MB 2025-02-14 05:44:15,187 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22187.87 MB 2025-02-14 05:44:15,187 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35527.85 MB 2025-02-14 05:44:15,187 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19885.41 MB 2025-02-14 05:44:15,453 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 05:44:15,453 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 05:44:15,453 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 05:44:15,453 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:44:15,453 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14851.24 MB 2025-02-14 05:44:15,453 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17862.32 MB 2025-02-14 05:44:15,453 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3011.08 MB 2025-02-14 05:44:15,453 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22187.87 MB 2025-02-14 05:44:15,453 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22187.87 MB 2025-02-14 05:44:15,453 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:44:15,453 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18163.39 MB 2025-02-14 05:44:15,471 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8154, cut from 8156 2025-02-14 05:44:15,471 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-14 05:44:15,477 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 05:44:15,477 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 05:44:15,477 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:44:15,477 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:44:15,477 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17862.32 MB 2025-02-14 05:44:15,477 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26293.00 MB 2025-02-14 05:44:15,477 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8430.68 MB 2025-02-14 05:44:15,477 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22187.87 MB 2025-02-14 05:44:15,477 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32667.34 MB 2025-02-14 05:44:15,477 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10479.47 MB 2025-02-14 05:44:15,477 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26293.00 MB 2025-02-14 05:44:15,628 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7946] 2025-02-14 05:44:15,630 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:44:15,630 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 05:44:15,631 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:44:15,631 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 05:44:15,635 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 05:44:15,636 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:44:15,636 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 05:44:15,636 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-14 05:45:05,797 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:45:05,797 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 05:45:05,802 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 05:45:05,806 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:45:05,806 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1369, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 05:45:05,807 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:45:05,807 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1369, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 05:45:26,818 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 05:45:26,818 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 05:45:26,818 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.00 seconds 2025-02-14 05:45:26,818 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:45:26,818 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22508.12 MB 2025-02-14 05:45:26,818 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27352.93 MB 2025-02-14 05:45:26,818 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4844.81 MB 2025-02-14 05:45:26,818 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45239.76 MB 2025-02-14 05:45:26,818 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38518.39 MB 2025-02-14 05:45:26,818 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6721.37 MB 2025-02-14 05:45:26,818 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36283.43 MB 2025-02-14 05:45:26,896 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 05:45:26,896 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 05:45:26,896 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 05:45:26,896 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:45:26,896 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27352.93 MB 2025-02-14 05:45:26,896 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22894.86 MB 2025-02-14 05:45:26,896 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4458.08 MB 2025-02-14 05:45:26,896 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38518.39 MB 2025-02-14 05:45:26,896 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47863.30 MB 2025-02-14 05:45:26,896 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9344.91 MB 2025-02-14 05:45:26,896 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41433.12 MB 2025-02-14 05:45:28,811 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 05:45:28,811 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 05:45:28,811 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 05:45:28,811 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:45:28,811 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22894.86 MB 2025-02-14 05:45:28,811 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23425.70 MB 2025-02-14 05:45:28,811 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 05:45:28,811 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47863.30 MB 2025-02-14 05:45:28,811 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33671.87 MB 2025-02-14 05:45:28,811 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14191.43 MB 2025-02-14 05:45:28,811 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27405.03 MB 2025-02-14 05:45:28,825 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 05:45:28,825 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 05:45:28,825 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 05:45:28,825 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:45:28,825 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23425.70 MB 2025-02-14 05:45:28,825 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25315.23 MB 2025-02-14 05:45:28,825 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 05:45:28,825 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33671.87 MB 2025-02-14 05:45:28,825 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33671.87 MB 2025-02-14 05:45:28,825 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:45:28,825 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26732.66 MB 2025-02-14 05:45:29,040 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 05:45:29,041 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 05:45:29,041 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 05:45:29,041 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:45:29,041 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25315.23 MB 2025-02-14 05:45:29,041 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27557.09 MB 2025-02-14 05:45:29,041 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 05:45:29,041 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33671.87 MB 2025-02-14 05:45:29,041 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37446.75 MB 2025-02-14 05:45:29,041 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 05:45:29,041 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33101.37 MB 2025-02-14 05:45:29,041 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 05:45:29,041 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 05:45:29,041 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 05:45:29,041 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:45:29,041 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23425.70 MB 2025-02-14 05:45:29,041 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27557.09 MB 2025-02-14 05:45:29,041 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 05:45:29,041 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33671.87 MB 2025-02-14 05:45:29,041 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37446.75 MB 2025-02-14 05:45:29,041 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 05:45:29,041 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33101.37 MB 2025-02-14 05:45:29,209 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 05:45:29,209 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 05:45:29,209 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 05:45:29,209 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:45:29,209 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29090.63 MB 2025-02-14 05:45:29,209 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29857.63 MB 2025-02-14 05:45:29,209 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 05:45:29,209 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37446.75 MB 2025-02-14 05:45:29,209 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37861.98 MB 2025-02-14 05:45:29,209 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 05:45:29,209 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30565.42 MB 2025-02-14 05:45:29,227 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 05:45:29,227 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 05:45:29,227 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:45:29,228 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:45:29,228 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30270.52 MB 2025-02-14 05:45:29,228 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30499.31 MB 2025-02-14 05:45:29,228 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.79 MB 2025-02-14 05:45:29,228 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37861.98 MB 2025-02-14 05:45:29,228 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37861.98 MB 2025-02-14 05:45:29,228 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:45:29,228 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30736.59 MB 2025-02-14 05:45:29,229 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 05:45:29,229 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 05:45:29,229 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.42 seconds 2025-02-14 05:45:29,229 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:45:29,229 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17738.41 MB 2025-02-14 05:45:29,229 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30700.01 MB 2025-02-14 05:45:29,229 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12961.60 MB 2025-02-14 05:45:29,229 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45239.76 MB 2025-02-14 05:45:29,229 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37861.98 MB 2025-02-14 05:45:29,229 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7377.78 MB 2025-02-14 05:45:29,229 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30736.59 MB 2025-02-14 05:45:29,497 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 05:45:29,497 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 05:45:29,497 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 05:45:29,497 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:45:29,497 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30700.01 MB 2025-02-14 05:45:29,497 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22737.09 MB 2025-02-14 05:45:29,497 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7962.93 MB 2025-02-14 05:45:29,497 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37861.98 MB 2025-02-14 05:45:29,497 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37861.98 MB 2025-02-14 05:45:29,497 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:45:29,497 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33207.07 MB 2025-02-14 05:45:29,514 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8147, cut from 8149 2025-02-14 05:45:29,515 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 05:45:29,521 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 05:45:29,521 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 05:45:29,521 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:45:29,521 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:45:29,521 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22737.09 MB 2025-02-14 05:45:29,521 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31160.29 MB 2025-02-14 05:45:29,521 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8423.21 MB 2025-02-14 05:45:29,521 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37861.98 MB 2025-02-14 05:45:29,521 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42049.99 MB 2025-02-14 05:45:29,521 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4188.01 MB 2025-02-14 05:45:29,521 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31160.29 MB 2025-02-14 05:45:29,684 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7939] 2025-02-14 05:45:29,685 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:45:29,685 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 05:45:29,686 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:45:29,686 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 05:45:29,691 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 05:45:29,692 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:45:29,692 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 05:45:29,692 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 05:47:52,681 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:47:52,681 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 05:47:52,687 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 05:47:52,691 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:47:52,692 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1160, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 05:47:52,692 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:47:52,692 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1160, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 05:48:10,446 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 05:48:10,446 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 05:48:10,446 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.75 seconds 2025-02-14 05:48:10,446 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:48:10,446 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21051.77 MB 2025-02-14 05:48:10,446 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25158.00 MB 2025-02-14 05:48:10,446 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4106.22 MB 2025-02-14 05:48:10,446 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50426.02 MB 2025-02-14 05:48:10,446 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29395.78 MB 2025-02-14 05:48:10,446 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21030.24 MB 2025-02-14 05:48:10,446 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34147.83 MB 2025-02-14 05:48:10,556 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 05:48:10,556 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 05:48:10,556 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 05:48:10,556 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:48:10,556 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25158.00 MB 2025-02-14 05:48:10,556 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21808.33 MB 2025-02-14 05:48:10,556 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3349.67 MB 2025-02-14 05:48:10,556 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29395.78 MB 2025-02-14 05:48:10,556 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43568.33 MB 2025-02-14 05:48:10,556 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 14172.55 MB 2025-02-14 05:48:10,557 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37081.16 MB 2025-02-14 05:48:12,459 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 05:48:12,459 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 05:48:12,459 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.90 seconds 2025-02-14 05:48:12,459 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:48:12,459 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21808.33 MB 2025-02-14 05:48:12,459 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22339.17 MB 2025-02-14 05:48:12,459 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 05:48:12,459 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43568.33 MB 2025-02-14 05:48:12,459 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26705.13 MB 2025-02-14 05:48:12,459 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16863.20 MB 2025-02-14 05:48:12,459 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26318.50 MB 2025-02-14 05:48:12,474 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 05:48:12,474 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 05:48:12,474 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 05:48:12,474 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:48:12,474 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22339.17 MB 2025-02-14 05:48:12,474 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24228.70 MB 2025-02-14 05:48:12,474 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 05:48:12,474 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26705.13 MB 2025-02-14 05:48:12,474 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28592.57 MB 2025-02-14 05:48:12,474 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 05:48:12,474 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25646.13 MB 2025-02-14 05:48:12,680 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 05:48:12,680 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 05:48:12,680 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 05:48:12,680 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:48:12,680 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24228.70 MB 2025-02-14 05:48:12,680 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26470.56 MB 2025-02-14 05:48:12,680 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 05:48:12,680 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28592.57 MB 2025-02-14 05:48:12,680 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34254.88 MB 2025-02-14 05:48:12,680 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 05:48:12,680 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32014.84 MB 2025-02-14 05:48:12,681 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 05:48:12,681 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 05:48:12,681 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 05:48:12,681 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:48:12,681 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22339.17 MB 2025-02-14 05:48:12,681 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26470.56 MB 2025-02-14 05:48:12,681 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 05:48:12,681 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26705.13 MB 2025-02-14 05:48:12,681 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34254.88 MB 2025-02-14 05:48:12,681 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-14 05:48:12,681 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32014.84 MB 2025-02-14 05:48:12,840 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 05:48:12,840 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 05:48:12,840 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 05:48:12,840 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:48:12,840 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28004.10 MB 2025-02-14 05:48:12,840 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28771.10 MB 2025-02-14 05:48:12,840 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 05:48:12,840 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34254.88 MB 2025-02-14 05:48:12,840 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34670.12 MB 2025-02-14 05:48:12,840 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 05:48:12,840 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29478.89 MB 2025-02-14 05:48:12,858 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 05:48:12,858 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 05:48:12,858 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:48:12,858 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:48:12,858 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29183.99 MB 2025-02-14 05:48:12,858 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29412.77 MB 2025-02-14 05:48:12,858 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.78 MB 2025-02-14 05:48:12,858 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34670.12 MB 2025-02-14 05:48:12,858 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34670.12 MB 2025-02-14 05:48:12,858 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:48:12,858 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29616.55 MB 2025-02-14 05:48:12,859 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 05:48:12,859 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 05:48:12,859 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.16 seconds 2025-02-14 05:48:12,859 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:48:12,859 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17010.24 MB 2025-02-14 05:48:12,859 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29613.45 MB 2025-02-14 05:48:12,859 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12603.21 MB 2025-02-14 05:48:12,859 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50426.02 MB 2025-02-14 05:48:12,859 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34670.12 MB 2025-02-14 05:48:12,859 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15755.90 MB 2025-02-14 05:48:12,859 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29616.55 MB 2025-02-14 05:48:13,125 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 05:48:13,125 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 05:48:13,125 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 05:48:13,125 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:48:13,125 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29613.45 MB 2025-02-14 05:48:13,125 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22008.53 MB 2025-02-14 05:48:13,125 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7604.92 MB 2025-02-14 05:48:13,125 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34670.12 MB 2025-02-14 05:48:13,125 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34670.12 MB 2025-02-14 05:48:13,125 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:48:13,125 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32120.20 MB 2025-02-14 05:48:13,143 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8146, cut from 8148 2025-02-14 05:48:13,143 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 05:48:13,149 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 05:48:13,149 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 05:48:13,149 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:48:13,149 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:48:13,149 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22008.53 MB 2025-02-14 05:48:13,149 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30430.86 MB 2025-02-14 05:48:13,149 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8422.33 MB 2025-02-14 05:48:13,149 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34670.12 MB 2025-02-14 05:48:13,149 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43044.04 MB 2025-02-14 05:48:13,149 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8373.93 MB 2025-02-14 05:48:13,149 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30430.86 MB 2025-02-14 05:48:13,302 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7938] 2025-02-14 05:48:13,303 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:48:13,303 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 05:48:13,304 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:48:13,304 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 05:48:13,309 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 05:48:13,310 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:48:13,310 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 05:48:13,310 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 05:49:09,716 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:49:09,716 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 05:49:09,721 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 05:49:09,725 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:49:09,725 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 3055, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 05:49:09,726 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:49:09,726 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 3055, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 05:49:57,111 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 05:49:57,111 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 05:49:57,111 - resource_logging.py:150 - __exit__ - DEBUG - Time: 47.37 seconds 2025-02-14 05:49:57,111 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:49:57,111 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34258.99 MB 2025-02-14 05:49:57,111 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45070.47 MB 2025-02-14 05:49:57,111 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 10811.47 MB 2025-02-14 05:49:57,111 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 76894.18 MB 2025-02-14 05:49:57,111 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49012.54 MB 2025-02-14 05:49:57,111 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27881.64 MB 2025-02-14 05:49:57,111 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 55881.94 MB 2025-02-14 05:49:57,318 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 05:49:57,319 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 05:49:57,319 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 05:49:57,319 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:49:57,319 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45070.47 MB 2025-02-14 05:49:57,319 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31661.68 MB 2025-02-14 05:49:57,319 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -13408.79 MB 2025-02-14 05:49:57,319 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49012.54 MB 2025-02-14 05:49:57,319 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 90779.42 MB 2025-02-14 05:49:57,319 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 41766.88 MB 2025-02-14 05:49:57,319 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 76885.19 MB 2025-02-14 05:49:59,332 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 05:49:59,332 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 05:49:59,332 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.01 seconds 2025-02-14 05:49:59,332 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:49:59,332 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31661.68 MB 2025-02-14 05:49:59,332 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32192.52 MB 2025-02-14 05:49:59,332 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 05:49:59,332 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 90779.42 MB 2025-02-14 05:49:59,332 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34208.74 MB 2025-02-14 05:49:59,332 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -56570.68 MB 2025-02-14 05:49:59,332 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36172.89 MB 2025-02-14 05:49:59,346 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 05:49:59,346 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 05:49:59,346 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 05:49:59,346 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:49:59,346 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32192.52 MB 2025-02-14 05:49:59,346 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34082.06 MB 2025-02-14 05:49:59,346 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 05:49:59,346 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34208.74 MB 2025-02-14 05:49:59,346 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37511.76 MB 2025-02-14 05:49:59,346 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 05:49:59,346 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35499.48 MB 2025-02-14 05:49:59,586 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 05:49:59,586 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 05:49:59,586 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.24 seconds 2025-02-14 05:49:59,586 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:49:59,586 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34082.06 MB 2025-02-14 05:49:59,586 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36323.91 MB 2025-02-14 05:49:59,586 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 05:49:59,586 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37511.76 MB 2025-02-14 05:49:59,586 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44117.79 MB 2025-02-14 05:49:59,586 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 05:49:59,586 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41868.19 MB 2025-02-14 05:49:59,588 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 05:49:59,588 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 05:49:59,588 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.25 seconds 2025-02-14 05:49:59,588 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:49:59,588 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32192.52 MB 2025-02-14 05:49:59,588 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36323.91 MB 2025-02-14 05:49:59,588 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 05:49:59,588 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34208.74 MB 2025-02-14 05:49:59,588 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44117.79 MB 2025-02-14 05:49:59,588 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9909.04 MB 2025-02-14 05:49:59,588 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41868.19 MB 2025-02-14 05:49:59,856 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 05:49:59,857 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 05:49:59,857 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 05:49:59,857 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:49:59,857 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37857.45 MB 2025-02-14 05:49:59,857 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38624.46 MB 2025-02-14 05:49:59,857 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 05:49:59,857 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44117.79 MB 2025-02-14 05:49:59,857 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44533.02 MB 2025-02-14 05:49:59,857 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 05:49:59,857 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39332.24 MB 2025-02-14 05:49:59,886 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 05:49:59,886 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 05:49:59,887 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 05:49:59,887 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:49:59,887 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39037.34 MB 2025-02-14 05:49:59,887 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39265.57 MB 2025-02-14 05:49:59,887 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.22 MB 2025-02-14 05:49:59,887 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44533.02 MB 2025-02-14 05:49:59,887 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44533.02 MB 2025-02-14 05:49:59,887 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:49:59,887 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39487.69 MB 2025-02-14 05:49:59,889 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 05:49:59,889 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 05:49:59,889 - resource_logging.py:150 - __exit__ - DEBUG - Time: 50.16 seconds 2025-02-14 05:49:59,889 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:49:59,889 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23613.85 MB 2025-02-14 05:49:59,889 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39465.71 MB 2025-02-14 05:49:59,889 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 15851.86 MB 2025-02-14 05:49:59,889 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 66249.03 MB 2025-02-14 05:49:59,889 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44533.02 MB 2025-02-14 05:49:59,889 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21716.01 MB 2025-02-14 05:49:59,889 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39487.69 MB 2025-02-14 05:50:00,178 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 05:50:00,178 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 05:50:00,178 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-14 05:50:00,178 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:50:00,178 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39465.71 MB 2025-02-14 05:50:00,178 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28603.76 MB 2025-02-14 05:50:00,178 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10861.94 MB 2025-02-14 05:50:00,178 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44533.02 MB 2025-02-14 05:50:00,178 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44533.02 MB 2025-02-14 05:50:00,178 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:50:00,178 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41965.70 MB 2025-02-14 05:50:00,196 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8124, cut from 8126 2025-02-14 05:50:00,196 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2 ('] 2025-02-14 05:50:00,202 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 05:50:00,202 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 05:50:00,202 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:50:00,202 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:50:00,202 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28603.76 MB 2025-02-14 05:50:00,202 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37003.67 MB 2025-02-14 05:50:00,202 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8399.91 MB 2025-02-14 05:50:00,202 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44533.02 MB 2025-02-14 05:50:00,202 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48708.45 MB 2025-02-14 05:50:00,203 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4175.43 MB 2025-02-14 05:50:00,203 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37003.67 MB 2025-02-14 05:50:00,391 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7916] 2025-02-14 05:50:00,392 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:50:00,392 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 05:50:00,393 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:50:00,393 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 05:50:00,398 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 05:50:00,399 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:50:00,399 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 05:50:00,399 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2 ('] 2025-02-14 05:50:52,580 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:50:52,581 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 05:50:52,586 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 05:50:52,590 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:50:52,590 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1264, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 05:50:52,590 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:50:52,591 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1264, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 05:51:12,100 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 05:51:12,100 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 05:51:12,100 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.50 seconds 2025-02-14 05:51:12,100 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:51:12,100 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21776.46 MB 2025-02-14 05:51:12,100 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26249.69 MB 2025-02-14 05:51:12,100 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4473.23 MB 2025-02-14 05:51:12,100 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57059.31 MB 2025-02-14 05:51:12,100 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39149.63 MB 2025-02-14 05:51:12,100 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17909.68 MB 2025-02-14 05:51:12,100 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35098.20 MB 2025-02-14 05:51:12,175 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 05:51:12,175 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 05:51:12,175 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 05:51:12,175 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:51:12,175 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26249.69 MB 2025-02-14 05:51:12,175 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22348.99 MB 2025-02-14 05:51:12,175 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3900.69 MB 2025-02-14 05:51:12,175 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39149.63 MB 2025-02-14 05:51:12,175 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45791.31 MB 2025-02-14 05:51:12,175 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6641.68 MB 2025-02-14 05:51:12,175 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39656.92 MB 2025-02-14 05:51:14,101 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 05:51:14,101 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 05:51:14,101 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 05:51:14,101 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:51:14,101 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22348.99 MB 2025-02-14 05:51:14,101 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22879.83 MB 2025-02-14 05:51:14,101 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 05:51:14,101 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45791.31 MB 2025-02-14 05:51:14,101 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30500.98 MB 2025-02-14 05:51:14,101 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15290.34 MB 2025-02-14 05:51:14,101 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26859.17 MB 2025-02-14 05:51:14,115 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 05:51:14,115 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 05:51:14,116 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 05:51:14,116 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:51:14,116 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22879.83 MB 2025-02-14 05:51:14,116 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24769.37 MB 2025-02-14 05:51:14,116 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 05:51:14,116 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30500.98 MB 2025-02-14 05:51:14,116 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30500.98 MB 2025-02-14 05:51:14,116 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:51:14,116 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26186.80 MB 2025-02-14 05:51:14,330 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 05:51:14,330 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 05:51:14,330 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 05:51:14,330 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:51:14,330 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24769.37 MB 2025-02-14 05:51:14,330 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27011.22 MB 2025-02-14 05:51:14,330 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 05:51:14,330 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30500.98 MB 2025-02-14 05:51:14,330 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35219.57 MB 2025-02-14 05:51:14,330 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 05:51:14,330 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32555.51 MB 2025-02-14 05:51:14,331 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 05:51:14,331 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 05:51:14,331 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 05:51:14,331 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:51:14,331 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22879.83 MB 2025-02-14 05:51:14,331 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27011.22 MB 2025-02-14 05:51:14,331 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 05:51:14,331 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30500.98 MB 2025-02-14 05:51:14,331 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35219.57 MB 2025-02-14 05:51:14,331 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 05:51:14,331 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32555.51 MB 2025-02-14 05:51:14,502 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 05:51:14,502 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 05:51:14,502 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 05:51:14,502 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:51:14,502 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28544.77 MB 2025-02-14 05:51:14,502 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29311.77 MB 2025-02-14 05:51:14,502 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 05:51:14,502 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35219.57 MB 2025-02-14 05:51:14,502 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35632.71 MB 2025-02-14 05:51:14,502 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 05:51:14,502 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30019.56 MB 2025-02-14 05:51:14,521 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 05:51:14,521 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 05:51:14,521 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:51:14,521 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:51:14,521 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29724.66 MB 2025-02-14 05:51:14,521 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29952.78 MB 2025-02-14 05:51:14,521 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.13 MB 2025-02-14 05:51:14,521 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35632.71 MB 2025-02-14 05:51:14,521 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35632.71 MB 2025-02-14 05:51:14,521 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:51:14,521 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30181.77 MB 2025-02-14 05:51:14,523 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 05:51:14,523 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 05:51:14,523 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.93 seconds 2025-02-14 05:51:14,523 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:51:14,523 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17372.58 MB 2025-02-14 05:51:14,523 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30153.73 MB 2025-02-14 05:51:14,523 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12781.15 MB 2025-02-14 05:51:14,523 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57059.31 MB 2025-02-14 05:51:14,523 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35632.71 MB 2025-02-14 05:51:14,523 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21426.60 MB 2025-02-14 05:51:14,523 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30181.77 MB 2025-02-14 05:51:14,793 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 05:51:14,793 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 05:51:14,793 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 05:51:14,793 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:51:14,793 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30153.73 MB 2025-02-14 05:51:14,793 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22375.07 MB 2025-02-14 05:51:14,793 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7778.66 MB 2025-02-14 05:51:14,793 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35632.71 MB 2025-02-14 05:51:14,793 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35632.71 MB 2025-02-14 05:51:14,793 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:51:14,793 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32663.86 MB 2025-02-14 05:51:14,811 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8157, cut from 8159 2025-02-14 05:51:14,811 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 05:51:14,817 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 05:51:14,817 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 05:51:14,817 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:51:14,817 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:51:14,817 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22375.07 MB 2025-02-14 05:51:14,817 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30808.89 MB 2025-02-14 05:51:14,817 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8433.82 MB 2025-02-14 05:51:14,817 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35632.71 MB 2025-02-14 05:51:14,817 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39824.92 MB 2025-02-14 05:51:14,817 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4192.21 MB 2025-02-14 05:51:14,817 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30808.89 MB 2025-02-14 05:51:14,986 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7949] 2025-02-14 05:51:14,987 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:51:14,987 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 05:51:14,988 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:51:14,988 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 05:51:14,993 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 05:51:14,994 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:51:14,994 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 05:51:14,994 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 05:52:45,388 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:52:45,388 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 05:52:45,393 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 05:52:45,397 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:52:45,397 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1340, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 05:52:45,398 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:52:45,398 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1340, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 05:53:05,965 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 05:53:05,965 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 05:53:05,965 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.56 seconds 2025-02-14 05:53:05,965 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:53:05,965 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22306.04 MB 2025-02-14 05:53:05,965 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27048.23 MB 2025-02-14 05:53:05,965 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4742.18 MB 2025-02-14 05:53:05,965 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48209.33 MB 2025-02-14 05:53:05,965 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38386.27 MB 2025-02-14 05:53:05,965 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9823.06 MB 2025-02-14 05:53:05,965 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35854.28 MB 2025-02-14 05:53:06,057 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 05:53:06,057 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 05:53:06,057 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 05:53:06,057 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:53:06,057 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27048.23 MB 2025-02-14 05:53:06,057 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22744.09 MB 2025-02-14 05:53:06,057 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4304.13 MB 2025-02-14 05:53:06,057 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38386.27 MB 2025-02-14 05:53:06,057 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41766.88 MB 2025-02-14 05:53:06,057 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3380.61 MB 2025-02-14 05:53:06,057 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38023.38 MB 2025-02-14 05:53:07,978 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 05:53:07,978 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 05:53:07,978 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 05:53:07,978 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:53:07,978 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22744.09 MB 2025-02-14 05:53:07,978 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23274.93 MB 2025-02-14 05:53:07,978 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 05:53:07,978 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41766.88 MB 2025-02-14 05:53:07,978 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29467.08 MB 2025-02-14 05:53:07,978 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12299.80 MB 2025-02-14 05:53:07,978 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27254.27 MB 2025-02-14 05:53:07,993 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 05:53:07,993 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 05:53:07,993 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 05:53:07,993 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:53:07,993 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23274.93 MB 2025-02-14 05:53:07,993 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25164.47 MB 2025-02-14 05:53:07,993 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 05:53:07,993 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29467.08 MB 2025-02-14 05:53:07,993 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30410.80 MB 2025-02-14 05:53:07,993 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 05:53:07,993 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26581.90 MB 2025-02-14 05:53:08,202 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 05:53:08,202 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 05:53:08,202 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 05:53:08,202 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:53:08,202 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25164.47 MB 2025-02-14 05:53:08,202 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27406.32 MB 2025-02-14 05:53:08,202 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 05:53:08,202 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30410.80 MB 2025-02-14 05:53:08,202 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35601.25 MB 2025-02-14 05:53:08,202 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 05:53:08,202 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32950.61 MB 2025-02-14 05:53:08,203 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 05:53:08,203 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 05:53:08,203 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 05:53:08,203 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:53:08,203 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23274.93 MB 2025-02-14 05:53:08,203 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27406.32 MB 2025-02-14 05:53:08,203 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 05:53:08,203 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29467.08 MB 2025-02-14 05:53:08,203 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35601.25 MB 2025-02-14 05:53:08,203 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 05:53:08,203 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32950.61 MB 2025-02-14 05:53:08,371 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 05:53:08,371 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 05:53:08,371 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 05:53:08,371 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:53:08,371 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28939.87 MB 2025-02-14 05:53:08,371 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29706.87 MB 2025-02-14 05:53:08,371 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 05:53:08,371 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35601.25 MB 2025-02-14 05:53:08,371 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36018.59 MB 2025-02-14 05:53:08,371 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 05:53:08,371 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30414.66 MB 2025-02-14 05:53:08,390 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 05:53:08,390 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 05:53:08,390 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:53:08,390 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:53:08,390 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30119.76 MB 2025-02-14 05:53:08,390 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30347.06 MB 2025-02-14 05:53:08,390 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.30 MB 2025-02-14 05:53:08,390 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36018.59 MB 2025-02-14 05:53:08,390 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36018.59 MB 2025-02-14 05:53:08,390 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:53:08,390 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30589.98 MB 2025-02-14 05:53:08,391 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 05:53:08,391 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 05:53:08,391 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.99 seconds 2025-02-14 05:53:08,391 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:53:08,391 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17637.37 MB 2025-02-14 05:53:08,391 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30547.81 MB 2025-02-14 05:53:08,391 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12910.44 MB 2025-02-14 05:53:08,391 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48209.33 MB 2025-02-14 05:53:08,392 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36018.59 MB 2025-02-14 05:53:08,392 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12190.74 MB 2025-02-14 05:53:08,392 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30589.98 MB 2025-02-14 05:53:08,660 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 05:53:08,660 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 05:53:08,660 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 05:53:08,660 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:53:08,660 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30547.81 MB 2025-02-14 05:53:08,660 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22636.81 MB 2025-02-14 05:53:08,660 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7911.00 MB 2025-02-14 05:53:08,660 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36018.59 MB 2025-02-14 05:53:08,660 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36018.59 MB 2025-02-14 05:53:08,660 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:53:08,660 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33055.48 MB 2025-02-14 05:53:08,678 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8149, cut from 8151 2025-02-14 05:53:08,678 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 05:53:08,684 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 05:53:08,684 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 05:53:08,684 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:53:08,684 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:53:08,684 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22636.81 MB 2025-02-14 05:53:08,684 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31062.99 MB 2025-02-14 05:53:08,684 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8426.18 MB 2025-02-14 05:53:08,684 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36018.59 MB 2025-02-14 05:53:08,685 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44394.61 MB 2025-02-14 05:53:08,685 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8376.03 MB 2025-02-14 05:53:08,685 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31062.99 MB 2025-02-14 05:53:08,847 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7941] 2025-02-14 05:53:08,849 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:53:08,849 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 05:53:08,850 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:53:08,850 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 05:53:08,854 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 05:53:08,855 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:53:08,855 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 05:53:08,856 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 05:54:55,771 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:54:55,772 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 05:54:55,777 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 05:54:55,781 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:54:55,781 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2079, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 05:54:55,782 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:54:55,782 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2079, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 05:55:27,755 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 05:55:27,755 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 05:55:27,755 - resource_logging.py:150 - __exit__ - DEBUG - Time: 31.96 seconds 2025-02-14 05:55:27,756 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:55:27,756 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27455.51 MB 2025-02-14 05:55:27,756 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34812.98 MB 2025-02-14 05:55:27,756 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7357.46 MB 2025-02-14 05:55:27,756 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52770.64 MB 2025-02-14 05:55:27,756 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41009.81 MB 2025-02-14 05:55:27,756 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11760.83 MB 2025-02-14 05:55:27,756 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43722.46 MB 2025-02-14 05:55:27,886 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 05:55:27,886 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 05:55:27,886 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 05:55:27,886 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:55:27,886 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34812.98 MB 2025-02-14 05:55:27,886 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26585.92 MB 2025-02-14 05:55:27,886 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8227.05 MB 2025-02-14 05:55:27,886 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41009.81 MB 2025-02-14 05:55:27,886 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 66714.60 MB 2025-02-14 05:55:27,887 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 25704.79 MB 2025-02-14 05:55:27,887 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 56362.40 MB 2025-02-14 05:55:29,839 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 05:55:29,839 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 05:55:29,840 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.95 seconds 2025-02-14 05:55:29,840 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:55:29,840 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26585.92 MB 2025-02-14 05:55:29,840 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27116.77 MB 2025-02-14 05:55:29,840 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 05:55:29,840 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 66714.60 MB 2025-02-14 05:55:29,840 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30891.05 MB 2025-02-14 05:55:29,840 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35823.55 MB 2025-02-14 05:55:29,840 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31097.14 MB 2025-02-14 05:55:29,855 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 05:55:29,855 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 05:55:29,855 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 05:55:29,855 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:55:29,855 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27116.77 MB 2025-02-14 05:55:29,855 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29006.30 MB 2025-02-14 05:55:29,855 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 05:55:29,855 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30891.05 MB 2025-02-14 05:55:29,855 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33722.20 MB 2025-02-14 05:55:29,855 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-14 05:55:29,855 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30423.73 MB 2025-02-14 05:55:30,064 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 05:55:30,064 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 05:55:30,064 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 05:55:30,064 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:55:30,064 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29006.30 MB 2025-02-14 05:55:30,064 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31248.16 MB 2025-02-14 05:55:30,064 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 05:55:30,064 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33722.20 MB 2025-02-14 05:55:30,064 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39384.51 MB 2025-02-14 05:55:30,064 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 05:55:30,064 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36792.44 MB 2025-02-14 05:55:30,065 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 05:55:30,065 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 05:55:30,065 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 05:55:30,065 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:55:30,065 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27116.77 MB 2025-02-14 05:55:30,065 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31248.16 MB 2025-02-14 05:55:30,065 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 05:55:30,065 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30891.05 MB 2025-02-14 05:55:30,065 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39384.51 MB 2025-02-14 05:55:30,065 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8493.47 MB 2025-02-14 05:55:30,065 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36792.44 MB 2025-02-14 05:55:30,231 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 05:55:30,231 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 05:55:30,231 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 05:55:30,231 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:55:30,231 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32781.70 MB 2025-02-14 05:55:30,231 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33548.70 MB 2025-02-14 05:55:30,231 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 05:55:30,231 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39384.51 MB 2025-02-14 05:55:30,231 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39797.65 MB 2025-02-14 05:55:30,231 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 05:55:30,231 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34256.49 MB 2025-02-14 05:55:30,250 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 05:55:30,250 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 05:55:30,250 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:55:30,250 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:55:30,250 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33961.59 MB 2025-02-14 05:55:30,250 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34190.27 MB 2025-02-14 05:55:30,250 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.69 MB 2025-02-14 05:55:30,250 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39797.65 MB 2025-02-14 05:55:30,250 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39797.65 MB 2025-02-14 05:55:30,250 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:55:30,250 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34406.01 MB 2025-02-14 05:55:30,251 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 05:55:30,251 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 05:55:30,251 - resource_logging.py:150 - __exit__ - DEBUG - Time: 34.47 seconds 2025-02-14 05:55:30,251 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:55:30,251 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20212.11 MB 2025-02-14 05:55:30,251 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34390.14 MB 2025-02-14 05:55:30,251 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14178.03 MB 2025-02-14 05:55:30,251 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52770.64 MB 2025-02-14 05:55:30,251 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39797.65 MB 2025-02-14 05:55:30,251 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12972.98 MB 2025-02-14 05:55:30,251 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34406.01 MB 2025-02-14 05:55:30,518 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 05:55:30,518 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 05:55:30,518 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 05:55:30,518 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:55:30,518 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34390.14 MB 2025-02-14 05:55:30,518 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25197.83 MB 2025-02-14 05:55:30,518 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9192.31 MB 2025-02-14 05:55:30,518 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39797.65 MB 2025-02-14 05:55:30,518 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39797.65 MB 2025-02-14 05:55:30,518 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:55:30,518 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36886.76 MB 2025-02-14 05:55:30,536 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8113, cut from 8115 2025-02-14 05:55:30,536 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 1 ('] 2025-02-14 05:55:30,542 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 05:55:30,542 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 05:55:30,542 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:55:30,542 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:55:30,542 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25197.83 MB 2025-02-14 05:55:30,542 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33586.25 MB 2025-02-14 05:55:30,542 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8388.42 MB 2025-02-14 05:55:30,542 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39797.65 MB 2025-02-14 05:55:30,542 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48138.03 MB 2025-02-14 05:55:30,542 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8340.37 MB 2025-02-14 05:55:30,542 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33586.25 MB 2025-02-14 05:55:30,704 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7905] 2025-02-14 05:55:30,705 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:55:30,705 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 05:55:30,706 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:55:30,706 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 05:55:30,711 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 05:55:30,712 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:55:30,712 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 05:55:30,712 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 1 ('] 2025-02-14 05:55:38,628 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:55:38,628 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 05:55:38,633 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 05:55:38,637 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:55:38,637 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2626, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 05:55:38,638 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:55:38,638 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2626, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 05:56:19,748 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 05:56:19,748 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 05:56:19,748 - resource_logging.py:150 - __exit__ - DEBUG - Time: 41.10 seconds 2025-02-14 05:56:19,748 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:56:19,748 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31268.45 MB 2025-02-14 05:56:19,748 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40561.72 MB 2025-02-14 05:56:19,748 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 9293.27 MB 2025-02-14 05:56:19,748 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 78947.29 MB 2025-02-14 05:56:19,748 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44503.66 MB 2025-02-14 05:56:19,748 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34443.62 MB 2025-02-14 05:56:19,748 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49856.02 MB 2025-02-14 05:56:19,916 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 05:56:19,916 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 05:56:19,916 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 05:56:19,916 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:56:19,916 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40561.72 MB 2025-02-14 05:56:19,916 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29430.31 MB 2025-02-14 05:56:19,916 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -11131.41 MB 2025-02-14 05:56:19,916 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44503.66 MB 2025-02-14 05:56:19,916 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 78756.45 MB 2025-02-14 05:56:19,916 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 34252.78 MB 2025-02-14 05:56:19,916 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 66735.14 MB 2025-02-14 05:56:21,891 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 05:56:21,891 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 05:56:21,891 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.97 seconds 2025-02-14 05:56:21,891 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:56:21,891 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29430.31 MB 2025-02-14 05:56:21,891 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29961.15 MB 2025-02-14 05:56:21,891 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 05:56:21,891 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 78756.45 MB 2025-02-14 05:56:21,891 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31977.37 MB 2025-02-14 05:56:21,891 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -46779.07 MB 2025-02-14 05:56:21,891 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33941.52 MB 2025-02-14 05:56:21,905 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 05:56:21,905 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 05:56:21,905 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 05:56:21,905 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:56:21,905 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29961.15 MB 2025-02-14 05:56:21,905 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31850.69 MB 2025-02-14 05:56:21,905 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 05:56:21,905 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31977.37 MB 2025-02-14 05:56:21,905 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35280.39 MB 2025-02-14 05:56:21,905 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 05:56:21,905 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33268.11 MB 2025-02-14 05:56:22,113 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 05:56:22,113 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 05:56:22,113 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 05:56:22,113 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:56:22,113 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31850.69 MB 2025-02-14 05:56:22,113 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34092.54 MB 2025-02-14 05:56:22,113 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 05:56:22,113 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35280.39 MB 2025-02-14 05:56:22,113 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41886.42 MB 2025-02-14 05:56:22,113 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 05:56:22,113 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39636.82 MB 2025-02-14 05:56:22,114 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 05:56:22,114 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 05:56:22,114 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 05:56:22,114 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:56:22,114 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29961.15 MB 2025-02-14 05:56:22,114 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34092.54 MB 2025-02-14 05:56:22,114 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 05:56:22,114 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31977.37 MB 2025-02-14 05:56:22,114 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41886.42 MB 2025-02-14 05:56:22,114 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9909.04 MB 2025-02-14 05:56:22,114 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39636.82 MB 2025-02-14 05:56:22,281 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 05:56:22,281 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 05:56:22,281 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 05:56:22,281 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:56:22,281 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35626.08 MB 2025-02-14 05:56:22,281 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36393.09 MB 2025-02-14 05:56:22,281 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 05:56:22,281 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41886.42 MB 2025-02-14 05:56:22,281 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42301.65 MB 2025-02-14 05:56:22,281 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 05:56:22,281 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37100.87 MB 2025-02-14 05:56:22,299 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 05:56:22,300 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 05:56:22,300 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:56:22,300 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:56:22,300 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36805.97 MB 2025-02-14 05:56:22,300 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37034.15 MB 2025-02-14 05:56:22,300 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.17 MB 2025-02-14 05:56:22,300 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42301.65 MB 2025-02-14 05:56:22,300 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42301.65 MB 2025-02-14 05:56:22,300 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:56:22,300 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37241.99 MB 2025-02-14 05:56:22,301 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 05:56:22,301 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 05:56:22,301 - resource_logging.py:150 - __exit__ - DEBUG - Time: 43.66 seconds 2025-02-14 05:56:22,301 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:56:22,301 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22118.58 MB 2025-02-14 05:56:22,301 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37234.24 MB 2025-02-14 05:56:22,301 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 15115.66 MB 2025-02-14 05:56:22,301 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 69797.41 MB 2025-02-14 05:56:22,301 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42301.65 MB 2025-02-14 05:56:22,301 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27495.76 MB 2025-02-14 05:56:22,301 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37241.99 MB 2025-02-14 05:56:22,573 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 05:56:22,573 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 05:56:22,573 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 05:56:22,573 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:56:22,573 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37234.24 MB 2025-02-14 05:56:22,573 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27107.73 MB 2025-02-14 05:56:22,573 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10126.51 MB 2025-02-14 05:56:22,573 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42301.65 MB 2025-02-14 05:56:22,573 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42301.65 MB 2025-02-14 05:56:22,573 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:56:22,573 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39733.62 MB 2025-02-14 05:56:22,591 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8122, cut from 8124 2025-02-14 05:56:22,591 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-14 05:56:22,597 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 05:56:22,597 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 05:56:22,597 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:56:22,597 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:56:22,597 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27107.73 MB 2025-02-14 05:56:22,597 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35505.14 MB 2025-02-14 05:56:22,597 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8397.40 MB 2025-02-14 05:56:22,597 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42301.65 MB 2025-02-14 05:56:22,597 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46477.08 MB 2025-02-14 05:56:22,597 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4175.43 MB 2025-02-14 05:56:22,597 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35505.14 MB 2025-02-14 05:56:22,757 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7914] 2025-02-14 05:56:22,759 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:56:22,759 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 05:56:22,760 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:56:22,760 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 05:56:22,764 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 05:56:22,765 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:56:22,765 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 05:56:22,766 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-14 05:57:08,247 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:57:08,247 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 05:57:08,252 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 05:57:08,256 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:57:08,256 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 152, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 05:57:08,257 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:57:08,257 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 152, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 05:57:10,641 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 05:57:10,641 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 05:57:10,641 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.38 seconds 2025-02-14 05:57:10,641 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:57:10,641 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14027.87 MB 2025-02-14 05:57:10,641 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14565.79 MB 2025-02-14 05:57:10,641 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 537.92 MB 2025-02-14 05:57:10,641 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54827.94 MB 2025-02-14 05:57:10,641 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 16909.34 MB 2025-02-14 05:57:10,641 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -37918.61 MB 2025-02-14 05:57:10,641 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23500.04 MB 2025-02-14 05:57:10,653 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 05:57:10,653 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 05:57:10,653 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 05:57:10,653 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:57:10,653 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14565.79 MB 2025-02-14 05:57:10,653 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14784.27 MB 2025-02-14 05:57:10,653 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 218.48 MB 2025-02-14 05:57:10,653 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 16909.34 MB 2025-02-14 05:57:10,653 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17678.99 MB 2025-02-14 05:57:10,654 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 769.65 MB 2025-02-14 05:57:10,654 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16644.89 MB 2025-02-14 05:57:11,364 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 05:57:11,364 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 05:57:11,364 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.71 seconds 2025-02-14 05:57:11,364 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:57:11,364 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14784.27 MB 2025-02-14 05:57:11,364 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14978.03 MB 2025-02-14 05:57:11,364 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 193.76 MB 2025-02-14 05:57:11,364 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17678.99 MB 2025-02-14 05:57:11,364 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17678.99 MB 2025-02-14 05:57:11,364 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:57:11,364 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18955.74 MB 2025-02-14 05:57:11,372 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 05:57:11,372 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 05:57:11,372 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 05:57:11,372 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:57:11,372 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14977.96 MB 2025-02-14 05:57:11,372 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15667.47 MB 2025-02-14 05:57:11,372 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 689.51 MB 2025-02-14 05:57:11,372 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17678.99 MB 2025-02-14 05:57:11,372 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17678.99 MB 2025-02-14 05:57:11,372 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:57:11,372 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16184.84 MB 2025-02-14 05:57:11,451 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 05:57:11,451 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 05:57:11,451 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 05:57:11,451 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:57:11,451 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15667.47 MB 2025-02-14 05:57:11,451 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16485.79 MB 2025-02-14 05:57:11,451 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 818.32 MB 2025-02-14 05:57:11,451 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17678.99 MB 2025-02-14 05:57:11,451 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19409.14 MB 2025-02-14 05:57:11,451 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1730.15 MB 2025-02-14 05:57:11,451 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18509.41 MB 2025-02-14 05:57:11,452 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 05:57:11,452 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 05:57:11,452 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 05:57:11,452 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:57:11,452 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14977.96 MB 2025-02-14 05:57:11,452 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16485.79 MB 2025-02-14 05:57:11,452 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1507.83 MB 2025-02-14 05:57:11,452 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17678.99 MB 2025-02-14 05:57:11,452 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19409.14 MB 2025-02-14 05:57:11,452 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1730.15 MB 2025-02-14 05:57:11,452 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18509.41 MB 2025-02-14 05:57:11,511 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 05:57:11,511 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 05:57:11,511 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 05:57:11,511 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:57:11,511 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17045.54 MB 2025-02-14 05:57:11,511 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17325.49 MB 2025-02-14 05:57:11,511 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 279.96 MB 2025-02-14 05:57:11,511 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19409.14 MB 2025-02-14 05:57:11,511 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19558.04 MB 2025-02-14 05:57:11,511 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 148.90 MB 2025-02-14 05:57:11,511 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17593.60 MB 2025-02-14 05:57:11,520 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 05:57:11,520 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 05:57:11,520 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 05:57:11,520 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:57:11,520 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17476.20 MB 2025-02-14 05:57:11,520 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17681.09 MB 2025-02-14 05:57:11,520 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 204.88 MB 2025-02-14 05:57:11,520 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19558.04 MB 2025-02-14 05:57:11,520 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19562.23 MB 2025-02-14 05:57:11,520 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-14 05:57:11,520 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17697.80 MB 2025-02-14 05:57:11,521 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 05:57:11,522 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 05:57:11,522 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.26 seconds 2025-02-14 05:57:11,522 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:57:11,522 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13498.29 MB 2025-02-14 05:57:11,522 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17881.96 MB 2025-02-14 05:57:11,522 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4383.68 MB 2025-02-14 05:57:11,522 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54827.94 MB 2025-02-14 05:57:11,522 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19562.23 MB 2025-02-14 05:57:11,522 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35265.71 MB 2025-02-14 05:57:11,522 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17881.96 MB 2025-02-14 05:57:11,787 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 05:57:11,787 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 05:57:11,788 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 05:57:11,788 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:57:11,788 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17881.96 MB 2025-02-14 05:57:11,788 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17300.60 MB 2025-02-14 05:57:11,788 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -581.36 MB 2025-02-14 05:57:11,788 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19562.23 MB 2025-02-14 05:57:11,788 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19830.67 MB 2025-02-14 05:57:11,788 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 268.44 MB 2025-02-14 05:57:11,788 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18986.41 MB 2025-02-14 05:57:11,806 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8154, cut from 8156 2025-02-14 05:57:11,806 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 05:57:11,812 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 05:57:11,812 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 05:57:11,812 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:57:11,812 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:57:11,812 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17300.60 MB 2025-02-14 05:57:11,812 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25732.01 MB 2025-02-14 05:57:11,812 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8431.41 MB 2025-02-14 05:57:11,812 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19830.67 MB 2025-02-14 05:57:11,812 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30310.14 MB 2025-02-14 05:57:11,812 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10479.47 MB 2025-02-14 05:57:11,812 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25732.01 MB 2025-02-14 05:57:11,967 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7946] 2025-02-14 05:57:11,968 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:57:11,968 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 05:57:11,969 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:57:11,969 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 05:57:11,974 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 05:57:11,975 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:57:11,975 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 05:57:11,975 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 05:57:24,319 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:57:24,319 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 05:57:24,325 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 05:57:24,328 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:57:24,328 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1069, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 05:57:24,329 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:57:24,329 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1069, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 05:57:40,881 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 05:57:40,881 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 05:57:40,881 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.55 seconds 2025-02-14 05:57:40,881 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:57:40,881 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20417.67 MB 2025-02-14 05:57:40,881 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24200.93 MB 2025-02-14 05:57:40,881 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3783.26 MB 2025-02-14 05:57:40,881 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42882.56 MB 2025-02-14 05:57:40,881 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31169.97 MB 2025-02-14 05:57:40,881 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11712.59 MB 2025-02-14 05:57:40,881 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33059.93 MB 2025-02-14 05:57:40,947 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 05:57:40,947 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 05:57:40,947 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 05:57:40,947 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:57:40,947 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24200.93 MB 2025-02-14 05:57:40,947 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21335.25 MB 2025-02-14 05:57:40,947 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2865.68 MB 2025-02-14 05:57:40,947 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31169.97 MB 2025-02-14 05:57:40,947 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40892.37 MB 2025-02-14 05:57:40,947 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9722.40 MB 2025-02-14 05:57:40,947 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35959.47 MB 2025-02-14 05:57:42,869 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 05:57:42,869 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 05:57:42,869 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 05:57:42,869 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:57:42,869 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21335.25 MB 2025-02-14 05:57:42,869 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21866.09 MB 2025-02-14 05:57:42,869 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 05:57:42,869 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40892.37 MB 2025-02-14 05:57:42,869 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28802.29 MB 2025-02-14 05:57:42,869 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12090.08 MB 2025-02-14 05:57:42,869 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25845.42 MB 2025-02-14 05:57:42,882 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 05:57:42,882 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 05:57:42,882 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 05:57:42,882 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:57:42,882 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21866.09 MB 2025-02-14 05:57:42,882 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23755.62 MB 2025-02-14 05:57:42,882 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 05:57:42,882 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28802.29 MB 2025-02-14 05:57:42,882 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28802.29 MB 2025-02-14 05:57:42,882 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:57:42,882 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25173.05 MB 2025-02-14 05:57:43,091 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 05:57:43,091 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 05:57:43,092 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 05:57:43,092 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:57:43,092 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23755.62 MB 2025-02-14 05:57:43,092 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25997.48 MB 2025-02-14 05:57:43,092 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 05:57:43,092 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28802.29 MB 2025-02-14 05:57:43,092 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34464.60 MB 2025-02-14 05:57:43,092 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 05:57:43,092 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31541.76 MB 2025-02-14 05:57:43,092 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 05:57:43,092 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 05:57:43,092 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 05:57:43,092 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:57:43,092 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21866.09 MB 2025-02-14 05:57:43,092 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25997.48 MB 2025-02-14 05:57:43,092 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 05:57:43,092 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28802.29 MB 2025-02-14 05:57:43,092 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34464.60 MB 2025-02-14 05:57:43,092 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 05:57:43,092 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31541.76 MB 2025-02-14 05:57:43,258 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 05:57:43,258 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 05:57:43,258 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 05:57:43,258 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:57:43,258 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27531.02 MB 2025-02-14 05:57:43,258 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28298.02 MB 2025-02-14 05:57:43,258 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 05:57:43,258 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34464.60 MB 2025-02-14 05:57:43,258 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34879.83 MB 2025-02-14 05:57:43,258 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 05:57:43,258 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29005.81 MB 2025-02-14 05:57:43,276 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 05:57:43,276 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 05:57:43,276 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:57:43,276 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:57:43,276 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28710.91 MB 2025-02-14 05:57:43,276 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28940.07 MB 2025-02-14 05:57:43,276 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.16 MB 2025-02-14 05:57:43,277 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34879.83 MB 2025-02-14 05:57:43,277 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34879.83 MB 2025-02-14 05:57:43,277 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:57:43,277 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29178.84 MB 2025-02-14 05:57:43,278 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 05:57:43,278 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 05:57:43,278 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.95 seconds 2025-02-14 05:57:43,278 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:57:43,278 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16693.19 MB 2025-02-14 05:57:43,278 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29140.45 MB 2025-02-14 05:57:43,278 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12447.27 MB 2025-02-14 05:57:43,278 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42882.56 MB 2025-02-14 05:57:43,278 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34879.83 MB 2025-02-14 05:57:43,278 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8002.73 MB 2025-02-14 05:57:43,278 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29178.84 MB 2025-02-14 05:57:43,547 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 05:57:43,547 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 05:57:43,547 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 05:57:43,547 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:57:43,547 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29140.45 MB 2025-02-14 05:57:43,547 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21686.91 MB 2025-02-14 05:57:43,547 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7453.54 MB 2025-02-14 05:57:43,547 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34879.83 MB 2025-02-14 05:57:43,547 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34879.83 MB 2025-02-14 05:57:43,547 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:57:43,547 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31643.52 MB 2025-02-14 05:57:43,565 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8134, cut from 8136 2025-02-14 05:57:43,565 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 05:57:43,571 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 05:57:43,571 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 05:57:43,571 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:57:43,571 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:57:43,571 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21686.91 MB 2025-02-14 05:57:43,571 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30096.72 MB 2025-02-14 05:57:43,571 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8409.81 MB 2025-02-14 05:57:43,571 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34879.83 MB 2025-02-14 05:57:43,571 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43241.18 MB 2025-02-14 05:57:43,571 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8361.35 MB 2025-02-14 05:57:43,571 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30096.72 MB 2025-02-14 05:57:43,733 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7926] 2025-02-14 05:57:43,735 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:57:43,735 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 05:57:43,735 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:57:43,736 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 05:57:43,740 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 05:57:43,741 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:57:43,741 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 05:57:43,741 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 05:58:14,224 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:58:14,224 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 05:58:14,229 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 05:58:14,232 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:58:14,232 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 222, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 05:58:14,233 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:58:14,233 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 222, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 05:58:17,689 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 05:58:17,689 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 05:58:17,689 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.45 seconds 2025-02-14 05:58:17,689 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:58:17,689 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14515.64 MB 2025-02-14 05:58:17,689 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15301.28 MB 2025-02-14 05:58:17,689 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 785.65 MB 2025-02-14 05:58:17,689 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55782.15 MB 2025-02-14 05:58:17,689 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17855.15 MB 2025-02-14 05:58:17,689 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -37926.99 MB 2025-02-14 05:58:17,689 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24214.31 MB 2025-02-14 05:58:17,706 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 05:58:17,706 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 05:58:17,706 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:58:17,706 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:58:17,706 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15301.28 MB 2025-02-14 05:58:17,706 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15682.65 MB 2025-02-14 05:58:17,706 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 381.36 MB 2025-02-14 05:58:17,706 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17855.15 MB 2025-02-14 05:58:17,706 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19809.70 MB 2025-02-14 05:58:17,706 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1954.55 MB 2025-02-14 05:58:17,706 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18471.66 MB 2025-02-14 05:58:18,776 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 05:58:18,776 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 05:58:18,776 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.07 seconds 2025-02-14 05:58:18,776 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:58:18,776 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15682.65 MB 2025-02-14 05:58:18,776 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15977.27 MB 2025-02-14 05:58:18,776 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 294.62 MB 2025-02-14 05:58:18,776 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19809.70 MB 2025-02-14 05:58:18,776 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18572.38 MB 2025-02-14 05:58:18,776 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1237.32 MB 2025-02-14 05:58:18,776 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19939.06 MB 2025-02-14 05:58:18,785 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 05:58:18,785 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 05:58:18,785 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 05:58:18,785 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:58:18,785 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15977.27 MB 2025-02-14 05:58:18,785 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17026.23 MB 2025-02-14 05:58:18,785 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1048.96 MB 2025-02-14 05:58:18,785 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18572.38 MB 2025-02-14 05:58:18,785 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19096.67 MB 2025-02-14 05:58:18,785 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 524.29 MB 2025-02-14 05:58:18,785 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17812.90 MB 2025-02-14 05:58:18,903 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 05:58:18,903 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 05:58:18,903 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 05:58:18,903 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:58:18,903 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17026.23 MB 2025-02-14 05:58:18,903 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18271.01 MB 2025-02-14 05:58:18,903 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1244.78 MB 2025-02-14 05:58:18,903 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19096.67 MB 2025-02-14 05:58:18,903 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22504.54 MB 2025-02-14 05:58:18,903 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3407.87 MB 2025-02-14 05:58:18,903 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21350.42 MB 2025-02-14 05:58:18,904 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 05:58:18,904 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 05:58:18,904 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 05:58:18,904 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:58:18,904 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15977.27 MB 2025-02-14 05:58:18,904 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18271.01 MB 2025-02-14 05:58:18,904 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2293.74 MB 2025-02-14 05:58:18,904 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18572.38 MB 2025-02-14 05:58:18,904 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22504.54 MB 2025-02-14 05:58:18,904 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3932.16 MB 2025-02-14 05:58:18,904 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21350.42 MB 2025-02-14 05:58:18,996 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 05:58:18,996 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 05:58:18,997 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 05:58:18,997 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:58:18,997 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19122.13 MB 2025-02-14 05:58:18,997 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19548.07 MB 2025-02-14 05:58:18,997 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 425.95 MB 2025-02-14 05:58:18,997 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22504.54 MB 2025-02-14 05:58:18,997 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22731.03 MB 2025-02-14 05:58:18,997 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 226.49 MB 2025-02-14 05:58:18,997 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19941.71 MB 2025-02-14 05:58:19,008 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 05:58:19,008 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 05:58:19,008 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 05:58:19,008 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:58:19,008 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19777.23 MB 2025-02-14 05:58:19,008 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19996.20 MB 2025-02-14 05:58:19,008 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 218.97 MB 2025-02-14 05:58:19,008 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22731.03 MB 2025-02-14 05:58:19,008 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22731.03 MB 2025-02-14 05:58:19,008 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:58:19,008 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20070.79 MB 2025-02-14 05:58:19,009 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 05:58:19,009 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 05:58:19,009 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.77 seconds 2025-02-14 05:58:19,009 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:58:19,009 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13742.17 MB 2025-02-14 05:58:19,009 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20196.66 MB 2025-02-14 05:58:19,009 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6454.48 MB 2025-02-14 05:58:19,009 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55782.15 MB 2025-02-14 05:58:19,009 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22731.03 MB 2025-02-14 05:58:19,010 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33051.12 MB 2025-02-14 05:58:19,010 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20196.66 MB 2025-02-14 05:58:19,277 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 05:58:19,277 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 05:58:19,277 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 05:58:19,277 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:58:19,277 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14892.19 MB 2025-02-14 05:58:19,277 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17897.00 MB 2025-02-14 05:58:19,277 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3004.82 MB 2025-02-14 05:58:19,277 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22731.03 MB 2025-02-14 05:58:19,277 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22731.03 MB 2025-02-14 05:58:19,277 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:58:19,277 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18197.45 MB 2025-02-14 05:58:19,295 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8137, cut from 8139 2025-02-14 05:58:19,295 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 05:58:19,301 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 05:58:19,301 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 05:58:19,302 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:58:19,302 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:58:19,302 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17897.00 MB 2025-02-14 05:58:19,302 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26310.53 MB 2025-02-14 05:58:19,302 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8413.52 MB 2025-02-14 05:58:19,302 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22731.03 MB 2025-02-14 05:58:19,302 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33185.33 MB 2025-02-14 05:58:19,302 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10454.30 MB 2025-02-14 05:58:19,302 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26310.53 MB 2025-02-14 05:58:19,467 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7929] 2025-02-14 05:58:19,468 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:58:19,468 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 05:58:19,469 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:58:19,469 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 05:58:19,474 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 05:58:19,475 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:58:19,475 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 05:58:19,475 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 05:58:29,604 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:58:29,604 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 05:58:29,609 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 05:58:29,612 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:58:29,612 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 650, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 05:58:29,613 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:58:29,613 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 650, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 05:58:39,710 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 05:58:39,710 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 05:58:39,710 - resource_logging.py:150 - __exit__ - DEBUG - Time: 10.09 seconds 2025-02-14 05:58:39,710 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:58:39,710 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17498.01 MB 2025-02-14 05:58:39,710 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19798.59 MB 2025-02-14 05:58:39,710 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2300.58 MB 2025-02-14 05:58:39,710 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41548.78 MB 2025-02-14 05:58:39,710 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25482.49 MB 2025-02-14 05:58:39,710 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16066.28 MB 2025-02-14 05:58:39,710 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28781.32 MB 2025-02-14 05:58:39,751 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 05:58:39,751 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 05:58:39,751 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-14 05:58:39,751 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:58:39,751 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19798.59 MB 2025-02-14 05:58:39,751 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19157.00 MB 2025-02-14 05:58:39,751 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -641.59 MB 2025-02-14 05:58:39,751 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25482.49 MB 2025-02-14 05:58:39,751 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31360.81 MB 2025-02-14 05:58:39,751 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5878.32 MB 2025-02-14 05:58:39,751 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28519.76 MB 2025-02-14 05:58:41,673 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 05:58:41,673 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 05:58:41,673 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 05:58:41,673 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:58:41,673 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19157.00 MB 2025-02-14 05:58:41,673 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19687.84 MB 2025-02-14 05:58:41,673 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 05:58:41,673 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31360.81 MB 2025-02-14 05:58:41,673 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24597.50 MB 2025-02-14 05:58:41,673 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6763.32 MB 2025-02-14 05:58:41,673 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23667.17 MB 2025-02-14 05:58:41,687 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 05:58:41,687 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 05:58:41,687 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 05:58:41,687 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:58:41,687 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19687.84 MB 2025-02-14 05:58:41,687 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21577.37 MB 2025-02-14 05:58:41,687 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 05:58:41,687 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24597.50 MB 2025-02-14 05:58:41,687 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25541.21 MB 2025-02-14 05:58:41,687 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 05:58:41,687 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22994.80 MB 2025-02-14 05:58:41,899 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 05:58:41,899 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 05:58:41,899 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 05:58:41,899 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:58:41,899 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21577.37 MB 2025-02-14 05:58:41,899 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23819.23 MB 2025-02-14 05:58:41,899 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 05:58:41,899 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25541.21 MB 2025-02-14 05:58:41,899 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31675.38 MB 2025-02-14 05:58:41,899 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 05:58:41,899 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29363.51 MB 2025-02-14 05:58:41,900 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 05:58:41,900 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 05:58:41,900 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 05:58:41,900 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:58:41,900 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19687.84 MB 2025-02-14 05:58:41,900 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23819.23 MB 2025-02-14 05:58:41,900 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 05:58:41,900 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24597.50 MB 2025-02-14 05:58:41,900 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31675.38 MB 2025-02-14 05:58:41,900 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7077.89 MB 2025-02-14 05:58:41,900 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29363.51 MB 2025-02-14 05:58:42,065 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 05:58:42,065 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 05:58:42,065 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 05:58:42,066 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:58:42,066 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25352.77 MB 2025-02-14 05:58:42,066 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26119.77 MB 2025-02-14 05:58:42,066 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 05:58:42,066 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31675.38 MB 2025-02-14 05:58:42,066 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32090.62 MB 2025-02-14 05:58:42,066 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 05:58:42,066 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26827.56 MB 2025-02-14 05:58:42,084 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 05:58:42,084 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 05:58:42,084 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:58:42,084 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:58:42,084 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26532.66 MB 2025-02-14 05:58:42,084 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26761.64 MB 2025-02-14 05:58:42,084 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.97 MB 2025-02-14 05:58:42,084 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32090.62 MB 2025-02-14 05:58:42,084 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32090.62 MB 2025-02-14 05:58:42,084 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:58:42,084 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26968.92 MB 2025-02-14 05:58:42,085 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 05:58:42,086 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 05:58:42,086 - resource_logging.py:150 - __exit__ - DEBUG - Time: 12.47 seconds 2025-02-14 05:58:42,086 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:58:42,086 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15233.36 MB 2025-02-14 05:58:42,086 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26962.71 MB 2025-02-14 05:58:42,086 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11729.35 MB 2025-02-14 05:58:42,086 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41548.78 MB 2025-02-14 05:58:42,086 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32090.62 MB 2025-02-14 05:58:42,086 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9458.16 MB 2025-02-14 05:58:42,086 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26968.92 MB 2025-02-14 05:58:42,355 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 05:58:42,355 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 05:58:42,355 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 05:58:42,355 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:58:42,355 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26962.71 MB 2025-02-14 05:58:42,355 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20237.75 MB 2025-02-14 05:58:42,355 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6724.96 MB 2025-02-14 05:58:42,355 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32090.62 MB 2025-02-14 05:58:42,355 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32090.62 MB 2025-02-14 05:58:42,355 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 05:58:42,355 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29474.38 MB 2025-02-14 05:58:42,373 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 05:58:42,374 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-14 05:58:42,380 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 05:58:42,380 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 05:58:42,380 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 05:58:42,380 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 05:58:42,380 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20237.75 MB 2025-02-14 05:58:42,380 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28676.77 MB 2025-02-14 05:58:42,380 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 05:58:42,380 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32090.62 MB 2025-02-14 05:58:42,380 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40481.33 MB 2025-02-14 05:58:42,380 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 05:58:42,380 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28676.77 MB 2025-02-14 05:58:42,543 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 05:58:42,544 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:58:42,544 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 05:58:42,545 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:58:42,545 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 05:58:42,550 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 05:58:42,551 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 05:58:42,551 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 05:58:42,551 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-14 06:00:04,262 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:00:04,262 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 06:00:04,267 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 06:00:04,271 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:00:04,271 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 179, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 06:00:04,272 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:00:04,273 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 179, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 06:00:07,026 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 06:00:07,026 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 06:00:07,026 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.75 seconds 2025-02-14 06:00:07,026 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:00:07,026 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14216.01 MB 2025-02-14 06:00:07,026 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14849.48 MB 2025-02-14 06:00:07,026 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 633.47 MB 2025-02-14 06:00:07,026 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53066.33 MB 2025-02-14 06:00:07,026 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17853.05 MB 2025-02-14 06:00:07,026 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35213.28 MB 2025-02-14 06:00:07,026 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23688.19 MB 2025-02-14 06:00:07,040 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 06:00:07,040 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 06:00:07,040 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:00:07,040 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:00:07,040 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14849.48 MB 2025-02-14 06:00:07,040 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15143.00 MB 2025-02-14 06:00:07,040 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 293.52 MB 2025-02-14 06:00:07,040 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17853.05 MB 2025-02-14 06:00:07,040 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18480.10 MB 2025-02-14 06:00:07,040 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 627.05 MB 2025-02-14 06:00:07,040 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17346.97 MB 2025-02-14 06:00:07,880 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 06:00:07,880 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 06:00:07,880 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.84 seconds 2025-02-14 06:00:07,880 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:00:07,880 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15143.00 MB 2025-02-14 06:00:07,880 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15377.90 MB 2025-02-14 06:00:07,881 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 234.90 MB 2025-02-14 06:00:07,881 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18480.10 MB 2025-02-14 06:00:07,881 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18316.53 MB 2025-02-14 06:00:07,881 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -163.58 MB 2025-02-14 06:00:07,881 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19313.88 MB 2025-02-14 06:00:07,889 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 06:00:07,889 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 06:00:07,889 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:00:07,889 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:00:07,889 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15377.84 MB 2025-02-14 06:00:07,889 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16213.75 MB 2025-02-14 06:00:07,889 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 835.92 MB 2025-02-14 06:00:07,889 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18316.53 MB 2025-02-14 06:00:07,889 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18316.53 MB 2025-02-14 06:00:07,889 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:00:07,889 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16840.97 MB 2025-02-14 06:00:07,983 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 06:00:07,983 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 06:00:07,983 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 06:00:07,983 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:00:07,983 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16213.75 MB 2025-02-14 06:00:07,983 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17205.81 MB 2025-02-14 06:00:07,983 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 992.06 MB 2025-02-14 06:00:07,983 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18316.53 MB 2025-02-14 06:00:07,983 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20833.11 MB 2025-02-14 06:00:07,983 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2516.58 MB 2025-02-14 06:00:07,983 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19659.12 MB 2025-02-14 06:00:07,984 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 06:00:07,984 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 06:00:07,984 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 06:00:07,984 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:00:07,984 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15377.84 MB 2025-02-14 06:00:07,984 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17205.81 MB 2025-02-14 06:00:07,984 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1827.97 MB 2025-02-14 06:00:07,984 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18316.53 MB 2025-02-14 06:00:07,984 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20833.11 MB 2025-02-14 06:00:07,984 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2516.58 MB 2025-02-14 06:00:07,984 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19659.12 MB 2025-02-14 06:00:08,065 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 06:00:08,066 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 06:00:08,066 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 06:00:08,066 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:00:08,066 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17884.40 MB 2025-02-14 06:00:08,066 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18223.80 MB 2025-02-14 06:00:08,066 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 339.40 MB 2025-02-14 06:00:08,066 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20833.11 MB 2025-02-14 06:00:08,066 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21015.56 MB 2025-02-14 06:00:08,066 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 182.45 MB 2025-02-14 06:00:08,066 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18544.22 MB 2025-02-14 06:00:08,075 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 06:00:08,075 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 06:00:08,075 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:00:08,075 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:00:08,075 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18406.51 MB 2025-02-14 06:00:08,075 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18633.34 MB 2025-02-14 06:00:08,075 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.83 MB 2025-02-14 06:00:08,075 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21015.56 MB 2025-02-14 06:00:08,075 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21015.56 MB 2025-02-14 06:00:08,075 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:00:08,075 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18662.66 MB 2025-02-14 06:00:08,077 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 06:00:08,077 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 06:00:08,077 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.80 seconds 2025-02-14 06:00:08,077 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:00:08,077 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13592.36 MB 2025-02-14 06:00:08,077 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18834.41 MB 2025-02-14 06:00:08,077 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5242.05 MB 2025-02-14 06:00:08,077 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53066.33 MB 2025-02-14 06:00:08,077 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21015.56 MB 2025-02-14 06:00:08,077 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32050.77 MB 2025-02-14 06:00:08,077 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18834.41 MB 2025-02-14 06:00:08,341 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 06:00:08,341 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 06:00:08,341 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 06:00:08,341 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:00:08,341 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18834.41 MB 2025-02-14 06:00:08,341 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17544.08 MB 2025-02-14 06:00:08,341 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1290.33 MB 2025-02-14 06:00:08,341 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21015.56 MB 2025-02-14 06:00:08,341 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21015.56 MB 2025-02-14 06:00:08,341 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:00:08,341 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19069.72 MB 2025-02-14 06:00:08,359 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 06:00:08,359 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2 ('] 2025-02-14 06:00:08,365 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 06:00:08,365 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 06:00:08,365 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:00:08,365 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:00:08,365 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17544.08 MB 2025-02-14 06:00:08,365 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25983.10 MB 2025-02-14 06:00:08,366 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 06:00:08,366 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21015.56 MB 2025-02-14 06:00:08,366 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31505.51 MB 2025-02-14 06:00:08,366 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 06:00:08,366 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25983.10 MB 2025-02-14 06:00:08,515 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 06:00:08,517 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:00:08,517 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 06:00:08,518 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:00:08,518 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 06:00:08,522 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 06:00:08,523 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:00:08,523 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 06:00:08,523 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2 ('] 2025-02-14 06:01:08,457 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:01:08,457 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 06:01:08,462 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 06:01:08,466 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:01:08,466 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1810, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 06:01:08,467 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:01:08,467 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1810, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 06:01:36,293 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 06:01:36,294 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 06:01:36,294 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.82 seconds 2025-02-14 06:01:36,294 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:01:36,294 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25581.08 MB 2025-02-14 06:01:36,294 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31986.57 MB 2025-02-14 06:01:36,294 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6405.49 MB 2025-02-14 06:01:36,294 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44090.52 MB 2025-02-14 06:01:36,294 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40097.55 MB 2025-02-14 06:01:36,294 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3992.98 MB 2025-02-14 06:01:36,294 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40942.06 MB 2025-02-14 06:01:36,401 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 06:01:36,401 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 06:01:36,401 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 06:01:36,401 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:01:36,401 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31986.57 MB 2025-02-14 06:01:36,401 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25187.48 MB 2025-02-14 06:01:36,401 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6799.09 MB 2025-02-14 06:01:36,401 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40097.55 MB 2025-02-14 06:01:36,401 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 59812.87 MB 2025-02-14 06:01:36,401 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 19715.33 MB 2025-02-14 06:01:36,401 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50543.71 MB 2025-02-14 06:01:38,328 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 06:01:38,328 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 06:01:38,328 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 06:01:38,328 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:01:38,328 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25187.48 MB 2025-02-14 06:01:38,328 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25718.32 MB 2025-02-14 06:01:38,328 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 06:01:38,328 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59812.87 MB 2025-02-14 06:01:38,328 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30912.02 MB 2025-02-14 06:01:38,328 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28900.85 MB 2025-02-14 06:01:38,328 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29697.65 MB 2025-02-14 06:01:38,345 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 06:01:38,345 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 06:01:38,345 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:01:38,345 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:01:38,345 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25718.32 MB 2025-02-14 06:01:38,345 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27607.85 MB 2025-02-14 06:01:38,345 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 06:01:38,345 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30912.02 MB 2025-02-14 06:01:38,345 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31855.74 MB 2025-02-14 06:01:38,345 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 06:01:38,345 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29025.28 MB 2025-02-14 06:01:38,553 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 06:01:38,553 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 06:01:38,553 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 06:01:38,554 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:01:38,554 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27607.85 MB 2025-02-14 06:01:38,554 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29849.71 MB 2025-02-14 06:01:38,554 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 06:01:38,554 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31855.74 MB 2025-02-14 06:01:38,554 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37989.91 MB 2025-02-14 06:01:38,554 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 06:01:38,554 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35393.99 MB 2025-02-14 06:01:38,554 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 06:01:38,554 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 06:01:38,554 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 06:01:38,554 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:01:38,554 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25718.32 MB 2025-02-14 06:01:38,554 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29849.71 MB 2025-02-14 06:01:38,554 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 06:01:38,554 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30912.02 MB 2025-02-14 06:01:38,554 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37989.91 MB 2025-02-14 06:01:38,554 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7077.89 MB 2025-02-14 06:01:38,554 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35393.99 MB 2025-02-14 06:01:38,730 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 06:01:38,730 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 06:01:38,730 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 06:01:38,730 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:01:38,731 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31383.25 MB 2025-02-14 06:01:38,731 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32150.25 MB 2025-02-14 06:01:38,731 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 06:01:38,731 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37989.91 MB 2025-02-14 06:01:38,731 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38403.05 MB 2025-02-14 06:01:38,731 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 06:01:38,731 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32858.04 MB 2025-02-14 06:01:38,749 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 06:01:38,749 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 06:01:38,749 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:01:38,749 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:01:38,749 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32563.14 MB 2025-02-14 06:01:38,750 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32791.92 MB 2025-02-14 06:01:38,750 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.78 MB 2025-02-14 06:01:38,750 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38403.05 MB 2025-02-14 06:01:38,750 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38403.05 MB 2025-02-14 06:01:38,750 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:01:38,750 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33025.10 MB 2025-02-14 06:01:38,751 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 06:01:38,751 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 06:01:38,751 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.28 seconds 2025-02-14 06:01:38,751 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:01:38,751 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19274.89 MB 2025-02-14 06:01:38,751 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32992.06 MB 2025-02-14 06:01:38,751 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13717.17 MB 2025-02-14 06:01:38,751 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44090.52 MB 2025-02-14 06:01:38,751 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38403.05 MB 2025-02-14 06:01:38,751 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5687.48 MB 2025-02-14 06:01:38,751 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33025.10 MB 2025-02-14 06:01:39,018 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 06:01:39,018 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 06:01:39,018 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 06:01:39,018 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:01:39,018 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32992.06 MB 2025-02-14 06:01:39,018 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24264.81 MB 2025-02-14 06:01:39,018 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8727.25 MB 2025-02-14 06:01:39,018 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38403.05 MB 2025-02-14 06:01:39,018 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38403.05 MB 2025-02-14 06:01:39,018 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:01:39,018 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35492.05 MB 2025-02-14 06:01:39,036 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8124, cut from 8126 2025-02-14 06:01:39,036 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 06:01:39,043 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 06:01:39,043 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 06:01:39,043 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:01:39,043 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:01:39,043 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24264.81 MB 2025-02-14 06:01:39,043 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32665.67 MB 2025-02-14 06:01:39,043 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8400.86 MB 2025-02-14 06:01:39,043 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38403.05 MB 2025-02-14 06:01:39,043 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46753.91 MB 2025-02-14 06:01:39,043 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8350.86 MB 2025-02-14 06:01:39,043 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32665.67 MB 2025-02-14 06:01:39,204 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7916] 2025-02-14 06:01:39,205 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:01:39,205 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 06:01:39,206 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:01:39,206 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 06:01:39,211 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 06:01:39,212 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:01:39,212 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 06:01:39,212 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 06:02:32,863 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:02:32,864 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 06:02:32,872 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 06:02:32,879 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:02:32,880 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1385, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 06:02:32,882 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:02:32,882 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1385, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 06:02:54,350 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 06:02:54,350 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 06:02:54,350 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.46 seconds 2025-02-14 06:02:54,350 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:02:54,350 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22619.61 MB 2025-02-14 06:02:54,350 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27521.05 MB 2025-02-14 06:02:54,350 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4901.44 MB 2025-02-14 06:02:54,350 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55104.77 MB 2025-02-14 06:02:54,350 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38514.20 MB 2025-02-14 06:02:54,350 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16590.57 MB 2025-02-14 06:02:54,350 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36394.33 MB 2025-02-14 06:02:54,430 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 06:02:54,430 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 06:02:54,430 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 06:02:54,431 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:02:54,431 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27521.05 MB 2025-02-14 06:02:54,431 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22978.03 MB 2025-02-14 06:02:54,431 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4543.01 MB 2025-02-14 06:02:54,431 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38514.20 MB 2025-02-14 06:02:54,431 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48228.20 MB 2025-02-14 06:02:54,431 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9714.01 MB 2025-02-14 06:02:54,431 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42161.31 MB 2025-02-14 06:02:56,351 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 06:02:56,351 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 06:02:56,351 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 06:02:56,351 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:02:56,351 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22978.03 MB 2025-02-14 06:02:56,351 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23508.88 MB 2025-02-14 06:02:56,351 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 06:02:56,351 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48228.20 MB 2025-02-14 06:02:56,351 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33611.06 MB 2025-02-14 06:02:56,351 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14617.15 MB 2025-02-14 06:02:56,351 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27488.21 MB 2025-02-14 06:02:56,364 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 06:02:56,364 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 06:02:56,364 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:02:56,364 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:02:56,364 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23508.88 MB 2025-02-14 06:02:56,365 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25398.41 MB 2025-02-14 06:02:56,365 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 06:02:56,365 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33611.06 MB 2025-02-14 06:02:56,365 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33611.06 MB 2025-02-14 06:02:56,365 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:02:56,365 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26815.84 MB 2025-02-14 06:02:56,575 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 06:02:56,575 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 06:02:56,575 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 06:02:56,576 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:02:56,576 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25398.41 MB 2025-02-14 06:02:56,576 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27640.27 MB 2025-02-14 06:02:56,576 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 06:02:56,576 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33611.06 MB 2025-02-14 06:02:56,576 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37385.93 MB 2025-02-14 06:02:56,576 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 06:02:56,576 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33184.55 MB 2025-02-14 06:02:56,576 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 06:02:56,576 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 06:02:56,576 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 06:02:56,576 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:02:56,576 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23508.88 MB 2025-02-14 06:02:56,576 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27640.27 MB 2025-02-14 06:02:56,576 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 06:02:56,576 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33611.06 MB 2025-02-14 06:02:56,576 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37385.93 MB 2025-02-14 06:02:56,576 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 06:02:56,576 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33184.55 MB 2025-02-14 06:02:56,744 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 06:02:56,744 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 06:02:56,744 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 06:02:56,744 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:02:56,744 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29173.81 MB 2025-02-14 06:02:56,744 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29940.81 MB 2025-02-14 06:02:56,744 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 06:02:56,744 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37385.93 MB 2025-02-14 06:02:56,744 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37801.16 MB 2025-02-14 06:02:56,744 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 06:02:56,744 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30648.60 MB 2025-02-14 06:02:56,763 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 06:02:56,763 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 06:02:56,763 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:02:56,763 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:02:56,763 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30353.70 MB 2025-02-14 06:02:56,763 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30582.73 MB 2025-02-14 06:02:56,763 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.03 MB 2025-02-14 06:02:56,763 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37801.16 MB 2025-02-14 06:02:56,763 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37801.16 MB 2025-02-14 06:02:56,763 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:02:56,763 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30825.25 MB 2025-02-14 06:02:56,764 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 06:02:56,764 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 06:02:56,764 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.88 seconds 2025-02-14 06:02:56,764 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:02:56,764 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17794.16 MB 2025-02-14 06:02:56,764 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30783.68 MB 2025-02-14 06:02:56,764 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12989.53 MB 2025-02-14 06:02:56,764 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55104.77 MB 2025-02-14 06:02:56,764 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37801.16 MB 2025-02-14 06:02:56,764 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17303.60 MB 2025-02-14 06:02:56,764 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30825.25 MB 2025-02-14 06:02:57,036 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 06:02:57,036 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 06:02:57,036 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 06:02:57,036 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:02:57,036 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30783.68 MB 2025-02-14 06:02:57,036 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22796.64 MB 2025-02-14 06:02:57,036 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7987.04 MB 2025-02-14 06:02:57,036 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37801.16 MB 2025-02-14 06:02:57,036 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37801.16 MB 2025-02-14 06:02:57,036 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:02:57,036 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33293.81 MB 2025-02-14 06:02:57,053 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8157, cut from 8159 2025-02-14 06:02:57,054 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 06:02:57,060 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 06:02:57,060 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 06:02:57,060 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:02:57,060 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:02:57,060 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22796.64 MB 2025-02-14 06:02:57,060 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31231.26 MB 2025-02-14 06:02:57,060 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8434.62 MB 2025-02-14 06:02:57,060 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37801.16 MB 2025-02-14 06:02:57,060 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46185.58 MB 2025-02-14 06:02:57,060 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8384.41 MB 2025-02-14 06:02:57,060 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31231.26 MB 2025-02-14 06:02:57,223 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7949] 2025-02-14 06:02:57,224 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:02:57,224 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 06:02:57,225 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:02:57,225 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 06:02:57,230 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 06:02:57,231 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:02:57,231 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 06:02:57,231 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 06:03:34,026 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:03:34,026 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 06:03:34,031 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 06:03:34,035 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:03:34,035 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1181, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 06:03:34,036 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:03:34,036 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1181, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 06:03:52,336 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 06:03:52,336 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 06:03:52,336 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.29 seconds 2025-02-14 06:03:52,336 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:03:52,336 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21198.10 MB 2025-02-14 06:03:52,336 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25377.73 MB 2025-02-14 06:03:52,336 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4179.62 MB 2025-02-14 06:03:52,336 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54569.99 MB 2025-02-14 06:03:52,336 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29439.82 MB 2025-02-14 06:03:52,336 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25130.17 MB 2025-02-14 06:03:52,336 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34294.16 MB 2025-02-14 06:03:52,416 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 06:03:52,416 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 06:03:52,418 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 06:03:52,418 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:03:52,418 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25377.73 MB 2025-02-14 06:03:52,418 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21917.50 MB 2025-02-14 06:03:52,418 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3460.23 MB 2025-02-14 06:03:52,418 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29439.82 MB 2025-02-14 06:03:52,418 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44975.52 MB 2025-02-14 06:03:52,418 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 15535.70 MB 2025-02-14 06:03:52,418 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37859.59 MB 2025-02-14 06:03:54,358 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 06:03:54,358 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 06:03:54,358 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-14 06:03:54,358 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:03:54,358 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21917.50 MB 2025-02-14 06:03:54,358 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22448.34 MB 2025-02-14 06:03:54,358 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 06:03:54,358 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44975.52 MB 2025-02-14 06:03:54,358 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26675.77 MB 2025-02-14 06:03:54,358 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18299.75 MB 2025-02-14 06:03:54,358 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26428.72 MB 2025-02-14 06:03:54,371 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 06:03:54,371 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 06:03:54,371 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:03:54,371 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:03:54,372 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22448.34 MB 2025-02-14 06:03:54,372 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24337.88 MB 2025-02-14 06:03:54,372 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 06:03:54,372 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26675.77 MB 2025-02-14 06:03:54,372 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28563.21 MB 2025-02-14 06:03:54,372 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 06:03:54,372 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25755.31 MB 2025-02-14 06:03:54,582 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 06:03:54,582 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 06:03:54,582 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 06:03:54,582 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:03:54,582 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24337.88 MB 2025-02-14 06:03:54,582 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26579.73 MB 2025-02-14 06:03:54,582 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 06:03:54,582 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28563.21 MB 2025-02-14 06:03:54,582 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34225.52 MB 2025-02-14 06:03:54,582 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 06:03:54,582 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32124.01 MB 2025-02-14 06:03:54,582 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 06:03:54,582 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 06:03:54,582 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 06:03:54,583 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:03:54,583 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22448.34 MB 2025-02-14 06:03:54,583 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26579.73 MB 2025-02-14 06:03:54,583 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 06:03:54,583 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26675.77 MB 2025-02-14 06:03:54,583 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34225.52 MB 2025-02-14 06:03:54,583 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-14 06:03:54,583 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32124.01 MB 2025-02-14 06:03:54,785 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 06:03:54,785 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 06:03:54,785 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 06:03:54,785 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:03:54,785 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28113.28 MB 2025-02-14 06:03:54,785 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28880.28 MB 2025-02-14 06:03:54,785 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 06:03:54,785 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34225.52 MB 2025-02-14 06:03:54,785 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34640.76 MB 2025-02-14 06:03:54,785 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 06:03:54,785 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29588.07 MB 2025-02-14 06:03:54,804 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 06:03:54,804 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 06:03:54,804 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:03:54,804 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:03:54,804 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29293.17 MB 2025-02-14 06:03:54,804 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29521.95 MB 2025-02-14 06:03:54,804 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.79 MB 2025-02-14 06:03:54,804 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34640.76 MB 2025-02-14 06:03:54,804 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34640.76 MB 2025-02-14 06:03:54,804 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:03:54,804 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29748.76 MB 2025-02-14 06:03:54,805 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 06:03:54,805 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 06:03:54,805 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.77 seconds 2025-02-14 06:03:54,805 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:03:54,805 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17083.40 MB 2025-02-14 06:03:54,805 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29722.29 MB 2025-02-14 06:03:54,805 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12638.89 MB 2025-02-14 06:03:54,805 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54569.99 MB 2025-02-14 06:03:54,805 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34640.76 MB 2025-02-14 06:03:54,805 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19929.24 MB 2025-02-14 06:03:54,805 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29748.76 MB 2025-02-14 06:03:55,078 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 06:03:55,078 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 06:03:55,078 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 06:03:55,078 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:03:55,078 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29722.29 MB 2025-02-14 06:03:55,078 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22076.37 MB 2025-02-14 06:03:55,078 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7645.92 MB 2025-02-14 06:03:55,078 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34640.76 MB 2025-02-14 06:03:55,078 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34640.76 MB 2025-02-14 06:03:55,078 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:03:55,078 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32224.74 MB 2025-02-14 06:03:55,096 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8132, cut from 8134 2025-02-14 06:03:55,096 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 06:03:55,102 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 06:03:55,102 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 06:03:55,102 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:03:55,102 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:03:55,102 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22076.37 MB 2025-02-14 06:03:55,102 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30485.66 MB 2025-02-14 06:03:55,102 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8409.30 MB 2025-02-14 06:03:55,102 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34640.76 MB 2025-02-14 06:03:55,102 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43000.00 MB 2025-02-14 06:03:55,102 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8359.25 MB 2025-02-14 06:03:55,102 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30485.66 MB 2025-02-14 06:03:55,264 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7924] 2025-02-14 06:03:55,266 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:03:55,266 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 06:03:55,267 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:03:55,267 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 06:03:55,272 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 06:03:55,273 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:03:55,273 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 06:03:55,273 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 06:05:20,980 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:05:20,980 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 06:05:20,985 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 06:05:20,989 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:05:20,989 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 646, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 06:05:20,990 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:05:20,990 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 646, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 06:05:30,935 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 06:05:30,935 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 06:05:30,935 - resource_logging.py:150 - __exit__ - DEBUG - Time: 9.94 seconds 2025-02-14 06:05:30,935 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:05:30,935 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17470.14 MB 2025-02-14 06:05:30,935 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19756.30 MB 2025-02-14 06:05:30,935 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2286.16 MB 2025-02-14 06:05:30,935 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51359.25 MB 2025-02-14 06:05:30,935 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24786.24 MB 2025-02-14 06:05:30,935 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26573.01 MB 2025-02-14 06:05:30,935 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28753.45 MB 2025-02-14 06:05:30,976 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 06:05:30,976 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 06:05:30,976 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-14 06:05:30,976 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:05:30,976 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19756.30 MB 2025-02-14 06:05:30,976 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19137.25 MB 2025-02-14 06:05:30,976 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -619.04 MB 2025-02-14 06:05:30,976 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24786.24 MB 2025-02-14 06:05:30,976 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31302.09 MB 2025-02-14 06:05:30,976 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6515.85 MB 2025-02-14 06:05:30,976 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28393.42 MB 2025-02-14 06:05:32,885 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 06:05:32,885 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 06:05:32,885 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 06:05:32,885 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:05:32,885 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19137.25 MB 2025-02-14 06:05:32,885 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19668.09 MB 2025-02-14 06:05:32,885 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 06:05:32,885 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31302.09 MB 2025-02-14 06:05:32,885 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24622.66 MB 2025-02-14 06:05:32,885 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6679.43 MB 2025-02-14 06:05:32,885 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23647.43 MB 2025-02-14 06:05:32,899 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 06:05:32,899 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 06:05:32,899 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:05:32,899 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:05:32,899 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19668.09 MB 2025-02-14 06:05:32,899 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21557.63 MB 2025-02-14 06:05:32,899 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 06:05:32,899 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24622.66 MB 2025-02-14 06:05:32,899 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25566.38 MB 2025-02-14 06:05:32,899 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 06:05:32,899 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22975.06 MB 2025-02-14 06:05:33,110 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 06:05:33,110 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 06:05:33,110 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 06:05:33,110 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:05:33,110 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21557.63 MB 2025-02-14 06:05:33,110 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23799.48 MB 2025-02-14 06:05:33,110 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 06:05:33,110 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25566.38 MB 2025-02-14 06:05:33,110 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31700.55 MB 2025-02-14 06:05:33,110 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 06:05:33,110 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29343.77 MB 2025-02-14 06:05:33,110 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 06:05:33,111 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 06:05:33,111 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 06:05:33,111 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:05:33,111 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19668.09 MB 2025-02-14 06:05:33,111 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23799.48 MB 2025-02-14 06:05:33,111 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 06:05:33,111 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24622.66 MB 2025-02-14 06:05:33,111 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31700.55 MB 2025-02-14 06:05:33,111 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7077.89 MB 2025-02-14 06:05:33,111 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29343.77 MB 2025-02-14 06:05:33,279 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 06:05:33,279 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 06:05:33,279 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 06:05:33,279 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:05:33,279 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25333.03 MB 2025-02-14 06:05:33,279 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26100.03 MB 2025-02-14 06:05:33,279 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 06:05:33,279 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31700.55 MB 2025-02-14 06:05:33,279 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32117.88 MB 2025-02-14 06:05:33,279 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 06:05:33,279 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26807.82 MB 2025-02-14 06:05:33,298 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 06:05:33,298 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 06:05:33,298 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:05:33,298 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:05:33,298 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26512.92 MB 2025-02-14 06:05:33,298 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26742.32 MB 2025-02-14 06:05:33,298 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.40 MB 2025-02-14 06:05:33,298 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32117.88 MB 2025-02-14 06:05:33,298 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32117.88 MB 2025-02-14 06:05:33,298 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:05:33,298 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26938.92 MB 2025-02-14 06:05:33,299 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 06:05:33,299 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 06:05:33,299 - resource_logging.py:150 - __exit__ - DEBUG - Time: 12.31 seconds 2025-02-14 06:05:33,299 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:05:33,299 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15219.42 MB 2025-02-14 06:05:33,300 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26943.39 MB 2025-02-14 06:05:33,300 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11723.97 MB 2025-02-14 06:05:33,300 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51359.25 MB 2025-02-14 06:05:33,300 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32117.88 MB 2025-02-14 06:05:33,300 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19241.37 MB 2025-02-14 06:05:33,300 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26943.39 MB 2025-02-14 06:05:33,567 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 06:05:33,567 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 06:05:33,567 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 06:05:33,567 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:05:33,567 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26943.39 MB 2025-02-14 06:05:33,567 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20223.81 MB 2025-02-14 06:05:33,567 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6719.58 MB 2025-02-14 06:05:33,567 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32117.88 MB 2025-02-14 06:05:33,567 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32117.88 MB 2025-02-14 06:05:33,567 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:05:33,567 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29455.06 MB 2025-02-14 06:05:33,585 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 06:05:33,585 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2 ('] 2025-02-14 06:05:33,592 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 06:05:33,592 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 06:05:33,592 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:05:33,592 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:05:33,592 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20223.81 MB 2025-02-14 06:05:33,592 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28662.83 MB 2025-02-14 06:05:33,592 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 06:05:33,592 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32117.88 MB 2025-02-14 06:05:33,592 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40508.59 MB 2025-02-14 06:05:33,592 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 06:05:33,592 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28662.83 MB 2025-02-14 06:05:33,755 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 06:05:33,757 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:05:33,757 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 06:05:33,758 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:05:33,758 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 06:05:33,762 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 06:05:33,763 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:05:33,763 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 06:05:33,764 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2 ('] 2025-02-14 06:06:58,615 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:06:58,616 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 06:06:58,621 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 06:06:58,626 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:06:58,626 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1960, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 06:06:58,628 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:06:58,628 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1960, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 06:07:28,791 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 06:07:28,791 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 06:07:28,791 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.15 seconds 2025-02-14 06:07:28,791 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:07:28,791 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26626.30 MB 2025-02-14 06:07:28,791 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33563.68 MB 2025-02-14 06:07:28,791 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6937.38 MB 2025-02-14 06:07:28,791 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53093.60 MB 2025-02-14 06:07:28,791 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40605.06 MB 2025-02-14 06:07:28,791 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12488.54 MB 2025-02-14 06:07:28,791 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42440.27 MB 2025-02-14 06:07:28,912 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 06:07:28,912 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 06:07:28,912 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 06:07:28,912 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:07:28,912 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33563.68 MB 2025-02-14 06:07:28,912 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25967.28 MB 2025-02-14 06:07:28,912 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7596.40 MB 2025-02-14 06:07:28,912 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40605.06 MB 2025-02-14 06:07:28,912 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 62606.28 MB 2025-02-14 06:07:28,912 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 22001.22 MB 2025-02-14 06:07:28,912 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 53049.35 MB 2025-02-14 06:07:30,841 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 06:07:30,841 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 06:07:30,841 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 06:07:30,841 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:07:30,841 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25967.28 MB 2025-02-14 06:07:30,841 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26498.12 MB 2025-02-14 06:07:30,841 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 06:07:30,841 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62606.28 MB 2025-02-14 06:07:30,841 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35083.26 MB 2025-02-14 06:07:30,841 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27523.02 MB 2025-02-14 06:07:30,841 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30477.45 MB 2025-02-14 06:07:30,854 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 06:07:30,854 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 06:07:30,854 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:07:30,854 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:07:30,854 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26498.12 MB 2025-02-14 06:07:30,854 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28387.65 MB 2025-02-14 06:07:30,854 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 06:07:30,854 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35083.26 MB 2025-02-14 06:07:30,854 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35083.26 MB 2025-02-14 06:07:30,854 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:07:30,854 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29805.08 MB 2025-02-14 06:07:31,068 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 06:07:31,068 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 06:07:31,068 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 06:07:31,068 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:07:31,068 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28387.65 MB 2025-02-14 06:07:31,068 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30629.51 MB 2025-02-14 06:07:31,068 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 06:07:31,068 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35083.26 MB 2025-02-14 06:07:31,068 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39801.85 MB 2025-02-14 06:07:31,068 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 06:07:31,068 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36173.79 MB 2025-02-14 06:07:31,069 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 06:07:31,069 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 06:07:31,069 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 06:07:31,069 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:07:31,069 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26498.12 MB 2025-02-14 06:07:31,069 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30629.51 MB 2025-02-14 06:07:31,069 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 06:07:31,069 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35083.26 MB 2025-02-14 06:07:31,069 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39801.85 MB 2025-02-14 06:07:31,069 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 06:07:31,069 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36173.79 MB 2025-02-14 06:07:31,237 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 06:07:31,237 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 06:07:31,237 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 06:07:31,237 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:07:31,237 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32163.05 MB 2025-02-14 06:07:31,237 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32930.05 MB 2025-02-14 06:07:31,237 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 06:07:31,237 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39801.85 MB 2025-02-14 06:07:31,237 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40217.08 MB 2025-02-14 06:07:31,237 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 06:07:31,237 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33637.84 MB 2025-02-14 06:07:31,256 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 06:07:31,256 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 06:07:31,256 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:07:31,256 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:07:31,256 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33342.94 MB 2025-02-14 06:07:31,256 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33571.72 MB 2025-02-14 06:07:31,256 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.78 MB 2025-02-14 06:07:31,256 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40217.08 MB 2025-02-14 06:07:31,256 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40217.08 MB 2025-02-14 06:07:31,256 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:07:31,256 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33759.44 MB 2025-02-14 06:07:31,257 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 06:07:31,257 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 06:07:31,257 - resource_logging.py:150 - __exit__ - DEBUG - Time: 32.63 seconds 2025-02-14 06:07:31,257 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:07:31,257 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19797.50 MB 2025-02-14 06:07:31,257 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33772.50 MB 2025-02-14 06:07:31,257 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13974.99 MB 2025-02-14 06:07:31,257 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53093.60 MB 2025-02-14 06:07:31,257 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40217.08 MB 2025-02-14 06:07:31,257 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12876.51 MB 2025-02-14 06:07:31,257 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33772.50 MB 2025-02-14 06:07:31,526 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 06:07:31,527 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 06:07:31,527 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 06:07:31,527 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:07:31,527 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33772.50 MB 2025-02-14 06:07:31,527 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24797.32 MB 2025-02-14 06:07:31,527 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8975.18 MB 2025-02-14 06:07:31,527 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40217.08 MB 2025-02-14 06:07:31,527 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40217.08 MB 2025-02-14 06:07:31,527 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:07:31,527 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36280.48 MB 2025-02-14 06:07:31,544 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8150, cut from 8152 2025-02-14 06:07:31,545 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2 ('] 2025-02-14 06:07:31,551 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 06:07:31,551 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 06:07:31,551 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:07:31,551 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:07:31,551 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24797.32 MB 2025-02-14 06:07:31,551 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33223.82 MB 2025-02-14 06:07:31,551 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8426.50 MB 2025-02-14 06:07:31,551 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40217.08 MB 2025-02-14 06:07:31,551 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48595.21 MB 2025-02-14 06:07:31,551 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8378.12 MB 2025-02-14 06:07:31,551 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33223.82 MB 2025-02-14 06:07:31,713 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7942] 2025-02-14 06:07:31,714 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:07:31,714 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 06:07:31,715 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:07:31,715 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 06:07:31,720 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 06:07:31,721 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:07:31,721 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 06:07:31,721 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2 ('] 2025-02-14 06:09:56,779 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:09:56,779 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 06:09:56,787 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 06:09:56,798 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:09:56,798 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1742, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 06:09:56,800 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:09:56,800 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1742, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 06:10:23,587 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 06:10:23,587 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 06:10:23,588 - resource_logging.py:150 - __exit__ - DEBUG - Time: 26.78 seconds 2025-02-14 06:10:23,588 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:10:23,588 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25107.24 MB 2025-02-14 06:10:23,588 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31272.87 MB 2025-02-14 06:10:23,588 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6165.63 MB 2025-02-14 06:10:23,588 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61161.34 MB 2025-02-14 06:10:23,588 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39816.53 MB 2025-02-14 06:10:23,588 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21344.81 MB 2025-02-14 06:10:23,588 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40241.73 MB 2025-02-14 06:10:23,691 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 06:10:23,691 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 06:10:23,691 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 06:10:23,691 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:10:23,691 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31272.87 MB 2025-02-14 06:10:23,691 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24833.97 MB 2025-02-14 06:10:23,691 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6438.90 MB 2025-02-14 06:10:23,691 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39816.53 MB 2025-02-14 06:10:23,691 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57388.56 MB 2025-02-14 06:10:23,691 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 17572.04 MB 2025-02-14 06:10:23,691 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48613.59 MB 2025-02-14 06:10:25,613 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 06:10:25,613 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 06:10:25,613 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 06:10:25,613 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:10:25,613 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24833.97 MB 2025-02-14 06:10:25,613 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25364.81 MB 2025-02-14 06:10:25,613 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 06:10:25,613 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57388.56 MB 2025-02-14 06:10:25,613 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35066.48 MB 2025-02-14 06:10:25,613 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22322.09 MB 2025-02-14 06:10:25,613 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29344.14 MB 2025-02-14 06:10:25,627 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 06:10:25,627 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 06:10:25,627 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:10:25,627 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:10:25,627 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25364.81 MB 2025-02-14 06:10:25,627 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27254.34 MB 2025-02-14 06:10:25,627 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 06:10:25,627 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35066.48 MB 2025-02-14 06:10:25,627 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35066.48 MB 2025-02-14 06:10:25,627 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:10:25,627 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28671.77 MB 2025-02-14 06:10:25,839 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 06:10:25,839 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 06:10:25,839 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 06:10:25,839 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:10:25,839 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27254.34 MB 2025-02-14 06:10:25,839 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29496.20 MB 2025-02-14 06:10:25,839 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 06:10:25,839 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35066.48 MB 2025-02-14 06:10:25,839 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38369.49 MB 2025-02-14 06:10:25,839 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 06:10:25,839 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35040.48 MB 2025-02-14 06:10:25,840 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 06:10:25,840 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 06:10:25,840 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 06:10:25,840 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:10:25,840 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25364.81 MB 2025-02-14 06:10:25,840 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29496.20 MB 2025-02-14 06:10:25,840 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 06:10:25,840 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35066.48 MB 2025-02-14 06:10:25,840 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38369.49 MB 2025-02-14 06:10:25,840 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 06:10:25,840 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35040.48 MB 2025-02-14 06:10:26,007 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 06:10:26,007 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 06:10:26,007 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 06:10:26,007 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:10:26,007 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31029.74 MB 2025-02-14 06:10:26,007 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31796.74 MB 2025-02-14 06:10:26,007 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 06:10:26,007 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38369.49 MB 2025-02-14 06:10:26,007 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38782.63 MB 2025-02-14 06:10:26,007 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 06:10:26,008 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32504.53 MB 2025-02-14 06:10:26,026 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 06:10:26,026 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 06:10:26,026 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:10:26,026 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:10:26,026 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32209.63 MB 2025-02-14 06:10:26,026 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32438.07 MB 2025-02-14 06:10:26,026 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.44 MB 2025-02-14 06:10:26,027 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38782.63 MB 2025-02-14 06:10:26,027 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38782.63 MB 2025-02-14 06:10:26,027 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:10:26,027 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32640.87 MB 2025-02-14 06:10:26,028 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 06:10:26,028 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 06:10:26,028 - resource_logging.py:150 - __exit__ - DEBUG - Time: 29.22 seconds 2025-02-14 06:10:26,028 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:10:26,028 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19037.97 MB 2025-02-14 06:10:26,028 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32638.43 MB 2025-02-14 06:10:26,028 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13600.46 MB 2025-02-14 06:10:26,028 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61161.34 MB 2025-02-14 06:10:26,028 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38782.63 MB 2025-02-14 06:10:26,028 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22378.71 MB 2025-02-14 06:10:26,028 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32640.87 MB 2025-02-14 06:10:26,295 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 06:10:26,295 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 06:10:26,295 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 06:10:26,296 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:10:26,296 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32638.43 MB 2025-02-14 06:10:26,296 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24031.32 MB 2025-02-14 06:10:26,296 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8607.12 MB 2025-02-14 06:10:26,296 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38782.63 MB 2025-02-14 06:10:26,296 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38782.63 MB 2025-02-14 06:10:26,296 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:10:26,296 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35141.19 MB 2025-02-14 06:10:26,313 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8133, cut from 8135 2025-02-14 06:10:26,313 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-14 06:10:26,320 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 06:10:26,320 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 06:10:26,320 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:10:26,320 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:10:26,320 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24031.32 MB 2025-02-14 06:10:26,320 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32440.60 MB 2025-02-14 06:10:26,320 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8409.29 MB 2025-02-14 06:10:26,320 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38782.63 MB 2025-02-14 06:10:26,320 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42962.26 MB 2025-02-14 06:10:26,320 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4179.62 MB 2025-02-14 06:10:26,320 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32440.60 MB 2025-02-14 06:10:26,482 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7925] 2025-02-14 06:10:26,484 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:10:26,484 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 06:10:26,485 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:10:26,485 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 06:10:26,489 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 06:10:26,490 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:10:26,490 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 06:10:26,491 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-14 06:11:31,195 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:11:31,195 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 06:11:31,200 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 06:11:31,204 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:11:31,205 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 3361, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 06:11:31,205 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:11:31,206 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 3361, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 06:12:23,375 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 06:12:23,375 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 06:12:23,375 - resource_logging.py:150 - __exit__ - DEBUG - Time: 52.16 seconds 2025-02-14 06:12:23,375 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:12:23,375 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36389.70 MB 2025-02-14 06:12:23,375 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48284.75 MB 2025-02-14 06:12:23,375 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11895.05 MB 2025-02-14 06:12:23,375 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 74742.50 MB 2025-02-14 06:12:23,375 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52225.38 MB 2025-02-14 06:12:23,375 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22517.12 MB 2025-02-14 06:12:23,375 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 60179.14 MB 2025-02-14 06:12:23,592 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 06:12:23,593 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 06:12:23,593 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 06:12:23,593 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:12:23,593 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 48284.75 MB 2025-02-14 06:12:23,593 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33251.32 MB 2025-02-14 06:12:23,593 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -15033.42 MB 2025-02-14 06:12:23,593 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52225.38 MB 2025-02-14 06:12:23,593 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 96821.31 MB 2025-02-14 06:12:23,593 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 44595.94 MB 2025-02-14 06:12:23,593 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 82116.21 MB 2025-02-14 06:12:25,582 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 06:12:25,582 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 06:12:25,582 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.99 seconds 2025-02-14 06:12:25,582 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:12:25,582 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33251.32 MB 2025-02-14 06:12:25,582 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33782.16 MB 2025-02-14 06:12:25,582 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 06:12:25,582 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 96821.31 MB 2025-02-14 06:12:25,582 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35798.38 MB 2025-02-14 06:12:25,582 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -61022.93 MB 2025-02-14 06:12:25,582 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37762.53 MB 2025-02-14 06:12:25,596 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 06:12:25,596 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 06:12:25,596 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:12:25,596 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:12:25,596 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33782.16 MB 2025-02-14 06:12:25,596 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35671.70 MB 2025-02-14 06:12:25,596 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 06:12:25,596 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35798.38 MB 2025-02-14 06:12:25,596 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39101.40 MB 2025-02-14 06:12:25,596 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 06:12:25,596 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37089.13 MB 2025-02-14 06:12:25,799 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 06:12:25,799 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 06:12:25,799 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 06:12:25,799 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:12:25,799 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35671.70 MB 2025-02-14 06:12:25,799 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37913.55 MB 2025-02-14 06:12:25,799 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 06:12:25,799 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39101.40 MB 2025-02-14 06:12:25,799 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45707.43 MB 2025-02-14 06:12:25,799 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 06:12:25,799 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43457.83 MB 2025-02-14 06:12:25,800 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 06:12:25,800 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 06:12:25,800 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 06:12:25,800 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:12:25,800 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33782.16 MB 2025-02-14 06:12:25,800 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37913.55 MB 2025-02-14 06:12:25,800 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 06:12:25,800 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35798.38 MB 2025-02-14 06:12:25,800 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45707.43 MB 2025-02-14 06:12:25,800 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9909.04 MB 2025-02-14 06:12:25,800 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43457.83 MB 2025-02-14 06:12:25,980 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 06:12:25,981 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 06:12:25,981 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-14 06:12:25,981 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:12:25,981 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39447.09 MB 2025-02-14 06:12:25,981 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40214.10 MB 2025-02-14 06:12:25,981 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 06:12:25,981 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45707.43 MB 2025-02-14 06:12:25,981 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46122.66 MB 2025-02-14 06:12:25,981 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 06:12:25,981 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40921.89 MB 2025-02-14 06:12:25,999 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 06:12:25,999 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 06:12:25,999 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:12:25,999 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:12:25,999 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40626.99 MB 2025-02-14 06:12:25,999 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40855.25 MB 2025-02-14 06:12:25,999 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.26 MB 2025-02-14 06:12:25,999 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46122.66 MB 2025-02-14 06:12:25,999 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46122.66 MB 2025-02-14 06:12:25,999 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:12:25,999 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41068.39 MB 2025-02-14 06:12:26,000 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 06:12:26,000 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 06:12:26,000 - resource_logging.py:150 - __exit__ - DEBUG - Time: 54.79 seconds 2025-02-14 06:12:26,000 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:12:26,000 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24679.20 MB 2025-02-14 06:12:26,000 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41055.48 MB 2025-02-14 06:12:26,000 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 16376.28 MB 2025-02-14 06:12:26,000 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63032.00 MB 2025-02-14 06:12:26,000 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46122.66 MB 2025-02-14 06:12:26,000 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16909.34 MB 2025-02-14 06:12:26,000 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41068.39 MB 2025-02-14 06:12:26,269 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 06:12:26,269 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 06:12:26,269 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 06:12:26,269 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:12:26,269 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41055.48 MB 2025-02-14 06:12:26,269 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29670.64 MB 2025-02-14 06:12:26,269 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -11384.84 MB 2025-02-14 06:12:26,269 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46122.66 MB 2025-02-14 06:12:26,269 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46122.66 MB 2025-02-14 06:12:26,269 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:12:26,269 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43556.71 MB 2025-02-14 06:12:26,287 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8128, cut from 8130 2025-02-14 06:12:26,287 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2 ('] 2025-02-14 06:12:26,293 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 06:12:26,293 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 06:12:26,293 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:12:26,293 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:12:26,293 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29670.64 MB 2025-02-14 06:12:26,293 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38074.72 MB 2025-02-14 06:12:26,293 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8404.08 MB 2025-02-14 06:12:26,293 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46122.66 MB 2025-02-14 06:12:26,293 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50300.19 MB 2025-02-14 06:12:26,293 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4177.53 MB 2025-02-14 06:12:26,293 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38074.72 MB 2025-02-14 06:12:26,444 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7920] 2025-02-14 06:12:26,445 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:12:26,445 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 06:12:26,446 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:12:26,446 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 06:12:26,450 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 06:12:26,451 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:12:26,451 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 06:12:26,451 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2 ('] 2025-02-14 06:13:28,718 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:13:28,718 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 06:13:28,725 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 06:13:28,730 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:13:28,730 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1197, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 06:13:28,732 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:13:28,732 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1197, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 06:13:47,302 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 06:13:47,302 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 06:13:47,302 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.56 seconds 2025-02-14 06:13:47,303 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:13:47,303 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21309.59 MB 2025-02-14 06:13:47,303 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25545.84 MB 2025-02-14 06:13:47,303 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4236.25 MB 2025-02-14 06:13:47,303 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58655.24 MB 2025-02-14 06:13:47,303 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29500.64 MB 2025-02-14 06:13:47,303 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29154.61 MB 2025-02-14 06:13:47,303 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34405.65 MB 2025-02-14 06:13:47,412 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 06:13:47,413 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 06:13:47,413 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 06:13:47,413 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:13:47,413 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25545.84 MB 2025-02-14 06:13:47,413 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22000.68 MB 2025-02-14 06:13:47,413 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3545.16 MB 2025-02-14 06:13:47,413 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29500.64 MB 2025-02-14 06:13:47,413 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44631.59 MB 2025-02-14 06:13:47,413 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 15130.95 MB 2025-02-14 06:13:47,413 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37781.92 MB 2025-02-14 06:13:49,371 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 06:13:49,371 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 06:13:49,371 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.96 seconds 2025-02-14 06:13:49,371 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:13:49,371 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22000.68 MB 2025-02-14 06:13:49,371 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22531.52 MB 2025-02-14 06:13:49,371 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 06:13:49,371 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44631.59 MB 2025-02-14 06:13:49,371 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26679.97 MB 2025-02-14 06:13:49,371 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17951.62 MB 2025-02-14 06:13:49,371 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26510.86 MB 2025-02-14 06:13:49,385 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 06:13:49,385 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 06:13:49,385 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:13:49,385 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:13:49,385 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22531.52 MB 2025-02-14 06:13:49,385 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24421.06 MB 2025-02-14 06:13:49,385 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 06:13:49,385 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26679.97 MB 2025-02-14 06:13:49,385 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28567.40 MB 2025-02-14 06:13:49,385 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 06:13:49,385 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25838.48 MB 2025-02-14 06:13:49,595 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 06:13:49,595 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 06:13:49,595 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 06:13:49,595 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:13:49,595 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24421.06 MB 2025-02-14 06:13:49,595 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26662.91 MB 2025-02-14 06:13:49,595 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 06:13:49,596 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28567.40 MB 2025-02-14 06:13:49,596 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34229.71 MB 2025-02-14 06:13:49,596 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 06:13:49,596 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32207.19 MB 2025-02-14 06:13:49,596 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 06:13:49,596 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 06:13:49,596 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 06:13:49,596 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:13:49,596 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22531.52 MB 2025-02-14 06:13:49,596 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26662.91 MB 2025-02-14 06:13:49,596 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 06:13:49,596 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26679.97 MB 2025-02-14 06:13:49,596 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34229.71 MB 2025-02-14 06:13:49,596 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-14 06:13:49,596 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32207.19 MB 2025-02-14 06:13:49,762 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 06:13:49,762 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 06:13:49,762 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 06:13:49,762 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:13:49,762 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28196.45 MB 2025-02-14 06:13:49,762 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28963.46 MB 2025-02-14 06:13:49,762 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 06:13:49,762 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34229.71 MB 2025-02-14 06:13:49,762 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34644.95 MB 2025-02-14 06:13:49,762 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 06:13:49,762 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29671.24 MB 2025-02-14 06:13:49,781 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 06:13:49,781 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 06:13:49,781 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:13:49,781 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:13:49,781 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29376.34 MB 2025-02-14 06:13:49,781 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29604.46 MB 2025-02-14 06:13:49,781 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.11 MB 2025-02-14 06:13:49,781 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34644.95 MB 2025-02-14 06:13:49,781 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34644.95 MB 2025-02-14 06:13:49,781 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:13:49,781 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29819.33 MB 2025-02-14 06:13:49,782 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 06:13:49,782 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 06:13:49,782 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.05 seconds 2025-02-14 06:13:49,782 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:13:49,782 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17139.15 MB 2025-02-14 06:13:49,782 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29805.51 MB 2025-02-14 06:13:49,782 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12666.36 MB 2025-02-14 06:13:49,782 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58655.24 MB 2025-02-14 06:13:49,782 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34644.95 MB 2025-02-14 06:13:49,782 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24010.29 MB 2025-02-14 06:13:49,782 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29819.33 MB 2025-02-14 06:13:50,052 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 06:13:50,052 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 06:13:50,052 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 06:13:50,052 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:13:50,052 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29805.51 MB 2025-02-14 06:13:50,052 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22143.16 MB 2025-02-14 06:13:50,052 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7662.35 MB 2025-02-14 06:13:50,052 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34644.95 MB 2025-02-14 06:13:50,052 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34644.95 MB 2025-02-14 06:13:50,052 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:13:50,052 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32316.87 MB 2025-02-14 06:13:50,070 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8161, cut from 8163 2025-02-14 06:13:50,070 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2 ('] 2025-02-14 06:13:50,076 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 06:13:50,076 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 06:13:50,076 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:13:50,076 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:13:50,076 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22143.16 MB 2025-02-14 06:13:50,076 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30582.00 MB 2025-02-14 06:13:50,076 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8438.84 MB 2025-02-14 06:13:50,076 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34644.95 MB 2025-02-14 06:13:50,076 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43033.56 MB 2025-02-14 06:13:50,077 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8388.61 MB 2025-02-14 06:13:50,077 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30582.00 MB 2025-02-14 06:13:50,242 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7953] 2025-02-14 06:13:50,244 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:13:50,244 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 06:13:50,245 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:13:50,245 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 06:13:50,250 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 06:13:50,251 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:13:50,251 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 06:13:50,251 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2 ('] 2025-02-14 06:14:00,168 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:14:00,168 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 06:14:00,173 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 06:14:00,177 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:14:00,177 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1209, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 06:14:00,178 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:14:00,178 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1209, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 06:14:19,052 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 06:14:19,052 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 06:14:19,052 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.87 seconds 2025-02-14 06:14:19,052 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:14:19,052 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21393.21 MB 2025-02-14 06:14:19,052 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25671.80 MB 2025-02-14 06:14:19,052 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4278.58 MB 2025-02-14 06:14:19,052 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51422.17 MB 2025-02-14 06:14:19,052 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33736.88 MB 2025-02-14 06:14:19,052 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17685.28 MB 2025-02-14 06:14:19,052 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34488.46 MB 2025-02-14 06:14:19,122 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 06:14:19,122 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 06:14:19,122 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 06:14:19,122 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:14:19,122 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25671.80 MB 2025-02-14 06:14:19,122 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22063.07 MB 2025-02-14 06:14:19,122 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3608.73 MB 2025-02-14 06:14:19,122 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33736.88 MB 2025-02-14 06:14:19,122 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42345.69 MB 2025-02-14 06:14:19,122 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8608.81 MB 2025-02-14 06:14:19,123 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38167.03 MB 2025-02-14 06:14:21,064 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 06:14:21,064 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 06:14:21,064 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-14 06:14:21,064 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:14:21,064 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22063.07 MB 2025-02-14 06:14:21,064 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22593.91 MB 2025-02-14 06:14:21,064 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 06:14:21,064 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42345.69 MB 2025-02-14 06:14:21,064 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25279.07 MB 2025-02-14 06:14:21,064 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17066.62 MB 2025-02-14 06:14:21,064 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26574.28 MB 2025-02-14 06:14:21,078 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 06:14:21,078 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 06:14:21,078 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:14:21,078 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:14:21,078 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22593.91 MB 2025-02-14 06:14:21,078 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24483.44 MB 2025-02-14 06:14:21,078 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 06:14:21,078 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25279.07 MB 2025-02-14 06:14:21,078 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28110.23 MB 2025-02-14 06:14:21,078 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-14 06:14:21,078 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25900.87 MB 2025-02-14 06:14:21,290 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 06:14:21,290 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 06:14:21,290 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 06:14:21,290 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:14:21,290 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24483.44 MB 2025-02-14 06:14:21,290 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26725.30 MB 2025-02-14 06:14:21,290 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 06:14:21,290 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28110.23 MB 2025-02-14 06:14:21,290 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34244.40 MB 2025-02-14 06:14:21,290 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 06:14:21,290 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32269.58 MB 2025-02-14 06:14:21,291 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 06:14:21,291 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 06:14:21,291 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 06:14:21,291 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:14:21,291 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22593.91 MB 2025-02-14 06:14:21,291 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26725.30 MB 2025-02-14 06:14:21,291 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 06:14:21,291 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25279.07 MB 2025-02-14 06:14:21,291 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34244.40 MB 2025-02-14 06:14:21,291 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8965.32 MB 2025-02-14 06:14:21,291 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32269.58 MB 2025-02-14 06:14:21,457 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 06:14:21,457 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 06:14:21,457 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 06:14:21,457 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:14:21,457 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28258.84 MB 2025-02-14 06:14:21,457 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29025.84 MB 2025-02-14 06:14:21,457 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 06:14:21,457 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34244.40 MB 2025-02-14 06:14:21,457 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34661.73 MB 2025-02-14 06:14:21,457 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 06:14:21,457 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29733.63 MB 2025-02-14 06:14:21,475 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 06:14:21,475 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 06:14:21,475 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:14:21,475 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:14:21,475 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29438.73 MB 2025-02-14 06:14:21,475 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29666.81 MB 2025-02-14 06:14:21,475 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.08 MB 2025-02-14 06:14:21,475 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34661.73 MB 2025-02-14 06:14:21,475 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34661.73 MB 2025-02-14 06:14:21,475 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:14:21,475 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29894.25 MB 2025-02-14 06:14:21,477 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 06:14:21,477 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 06:14:21,477 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.30 seconds 2025-02-14 06:14:21,477 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:14:21,477 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17180.96 MB 2025-02-14 06:14:21,477 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29866.80 MB 2025-02-14 06:14:21,477 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12685.84 MB 2025-02-14 06:14:21,477 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51422.17 MB 2025-02-14 06:14:21,477 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34661.73 MB 2025-02-14 06:14:21,477 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16760.44 MB 2025-02-14 06:14:21,477 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29894.25 MB 2025-02-14 06:14:21,751 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 06:14:21,751 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 06:14:21,751 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 06:14:21,751 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:14:21,751 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29866.80 MB 2025-02-14 06:14:21,751 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22168.59 MB 2025-02-14 06:14:21,751 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7698.21 MB 2025-02-14 06:14:21,751 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34661.73 MB 2025-02-14 06:14:21,751 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34661.73 MB 2025-02-14 06:14:21,751 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:14:21,751 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32364.95 MB 2025-02-14 06:14:21,769 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8118, cut from 8120 2025-02-14 06:14:21,769 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 06:14:21,775 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 06:14:21,775 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 06:14:21,775 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:14:21,775 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:14:21,775 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22168.59 MB 2025-02-14 06:14:21,775 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30561.86 MB 2025-02-14 06:14:21,775 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8393.27 MB 2025-02-14 06:14:21,775 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34661.73 MB 2025-02-14 06:14:21,775 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43008.39 MB 2025-02-14 06:14:21,775 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8346.66 MB 2025-02-14 06:14:21,775 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30561.86 MB 2025-02-14 06:14:21,938 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7910] 2025-02-14 06:14:21,939 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:14:21,939 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 06:14:21,940 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:14:21,940 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 06:14:21,945 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 06:14:21,946 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:14:21,946 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 06:14:21,946 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 06:14:30,900 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:14:30,900 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 06:14:30,905 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 06:14:30,909 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:14:30,909 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 134, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 06:14:30,910 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:14:30,910 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 134, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 06:14:33,052 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 06:14:33,052 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 06:14:33,052 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.14 seconds 2025-02-14 06:14:33,052 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:14:33,052 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13902.44 MB 2025-02-14 06:14:33,052 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14376.66 MB 2025-02-14 06:14:33,052 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 474.22 MB 2025-02-14 06:14:33,052 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51355.06 MB 2025-02-14 06:14:33,052 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 16909.34 MB 2025-02-14 06:14:33,052 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34445.72 MB 2025-02-14 06:14:33,052 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23374.62 MB 2025-02-14 06:14:33,063 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 06:14:33,063 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 06:14:33,063 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:14:33,063 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:14:33,063 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14376.66 MB 2025-02-14 06:14:33,063 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14607.35 MB 2025-02-14 06:14:33,063 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 230.69 MB 2025-02-14 06:14:33,063 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 16909.34 MB 2025-02-14 06:14:33,063 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17616.08 MB 2025-02-14 06:14:33,063 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 706.74 MB 2025-02-14 06:14:33,063 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16281.42 MB 2025-02-14 06:14:33,724 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 06:14:33,724 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 06:14:33,724 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.66 seconds 2025-02-14 06:14:33,724 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:14:33,724 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14607.35 MB 2025-02-14 06:14:33,724 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14785.18 MB 2025-02-14 06:14:33,724 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 177.83 MB 2025-02-14 06:14:33,724 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17616.08 MB 2025-02-14 06:14:33,724 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17616.08 MB 2025-02-14 06:14:33,724 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:14:33,724 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18778.82 MB 2025-02-14 06:14:33,731 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 06:14:33,731 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 06:14:33,731 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 06:14:33,731 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:14:33,731 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14785.11 MB 2025-02-14 06:14:33,731 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15417.95 MB 2025-02-14 06:14:33,731 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 632.84 MB 2025-02-14 06:14:33,731 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17616.08 MB 2025-02-14 06:14:33,731 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17616.08 MB 2025-02-14 06:14:33,731 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:14:33,731 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15892.80 MB 2025-02-14 06:14:33,807 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 06:14:33,807 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 06:14:33,807 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 06:14:33,807 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:14:33,807 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15417.95 MB 2025-02-14 06:14:33,807 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16169.02 MB 2025-02-14 06:14:33,807 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 751.06 MB 2025-02-14 06:14:33,807 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17616.08 MB 2025-02-14 06:14:33,807 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18884.85 MB 2025-02-14 06:14:33,807 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1268.78 MB 2025-02-14 06:14:33,807 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18027.88 MB 2025-02-14 06:14:33,808 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 06:14:33,808 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 06:14:33,808 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 06:14:33,808 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:14:33,808 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14785.11 MB 2025-02-14 06:14:33,808 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16169.02 MB 2025-02-14 06:14:33,808 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1383.90 MB 2025-02-14 06:14:33,808 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17616.08 MB 2025-02-14 06:14:33,808 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18884.85 MB 2025-02-14 06:14:33,808 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1268.78 MB 2025-02-14 06:14:33,808 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18027.88 MB 2025-02-14 06:14:33,865 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 06:14:33,865 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 06:14:33,865 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-14 06:14:33,865 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:14:33,865 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16682.75 MB 2025-02-14 06:14:33,865 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16939.70 MB 2025-02-14 06:14:33,865 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 256.95 MB 2025-02-14 06:14:33,865 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18884.85 MB 2025-02-14 06:14:33,865 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19021.17 MB 2025-02-14 06:14:33,865 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 136.31 MB 2025-02-14 06:14:33,865 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17188.53 MB 2025-02-14 06:14:33,873 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 06:14:33,873 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 06:14:33,873 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:14:33,873 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:14:33,873 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17078.03 MB 2025-02-14 06:14:33,873 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17298.10 MB 2025-02-14 06:14:33,873 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 220.07 MB 2025-02-14 06:14:33,873 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19021.17 MB 2025-02-14 06:14:33,873 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19021.17 MB 2025-02-14 06:14:33,873 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:14:33,873 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17299.58 MB 2025-02-14 06:14:33,875 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 06:14:33,875 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 06:14:33,875 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.96 seconds 2025-02-14 06:14:33,875 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:14:33,875 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13435.57 MB 2025-02-14 06:14:33,875 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14169.87 MB 2025-02-14 06:14:33,875 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 734.29 MB 2025-02-14 06:14:33,875 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51355.06 MB 2025-02-14 06:14:33,875 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19021.17 MB 2025-02-14 06:14:33,875 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32333.89 MB 2025-02-14 06:14:33,875 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17498.80 MB 2025-02-14 06:14:34,143 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 06:14:34,144 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 06:14:34,144 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 06:14:34,144 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:14:34,144 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14169.87 MB 2025-02-14 06:14:34,144 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17178.37 MB 2025-02-14 06:14:34,144 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3008.50 MB 2025-02-14 06:14:34,144 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19021.17 MB 2025-02-14 06:14:34,144 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19021.17 MB 2025-02-14 06:14:34,144 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:14:34,144 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17479.18 MB 2025-02-14 06:14:34,162 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8147, cut from 8149 2025-02-14 06:14:34,162 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 1 ('] 2025-02-14 06:14:34,168 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 06:14:34,168 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 06:14:34,168 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:14:34,168 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:14:34,168 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17178.37 MB 2025-02-14 06:14:34,168 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25601.58 MB 2025-02-14 06:14:34,168 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8423.21 MB 2025-02-14 06:14:34,168 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19021.17 MB 2025-02-14 06:14:34,168 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29492.25 MB 2025-02-14 06:14:34,168 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10471.08 MB 2025-02-14 06:14:34,168 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25601.58 MB 2025-02-14 06:14:34,330 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7939] 2025-02-14 06:14:34,332 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:14:34,332 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 06:14:34,333 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:14:34,333 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 06:14:34,338 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 06:14:34,339 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:14:34,339 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 06:14:34,339 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 1 ('] 2025-02-14 06:14:44,074 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:14:44,074 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 06:14:44,079 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 06:14:44,082 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:14:44,082 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 175, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 06:14:44,083 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:14:44,083 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 175, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 06:14:46,822 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 06:14:46,822 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 06:14:46,822 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.74 seconds 2025-02-14 06:14:46,822 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:14:46,822 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19523.70 MB 2025-02-14 06:14:46,822 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20143.02 MB 2025-02-14 06:14:46,822 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 619.32 MB 2025-02-14 06:14:46,822 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37868.27 MB 2025-02-14 06:14:46,822 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22366.13 MB 2025-02-14 06:14:46,822 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15502.15 MB 2025-02-14 06:14:46,822 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28995.88 MB 2025-02-14 06:14:46,837 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 06:14:46,837 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 06:14:46,837 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:14:46,837 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:14:46,837 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20143.02 MB 2025-02-14 06:14:46,837 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20443.01 MB 2025-02-14 06:14:46,837 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 299.99 MB 2025-02-14 06:14:46,837 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22366.13 MB 2025-02-14 06:14:46,837 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23909.63 MB 2025-02-14 06:14:46,837 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1543.50 MB 2025-02-14 06:14:46,837 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22654.16 MB 2025-02-14 06:14:47,694 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 06:14:47,695 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 06:14:47,695 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.86 seconds 2025-02-14 06:14:47,695 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:14:47,695 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20443.01 MB 2025-02-14 06:14:47,695 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20675.25 MB 2025-02-14 06:14:47,695 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 232.24 MB 2025-02-14 06:14:47,695 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23909.63 MB 2025-02-14 06:14:47,695 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23584.57 MB 2025-02-14 06:14:47,695 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -325.06 MB 2025-02-14 06:14:47,695 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24614.48 MB 2025-02-14 06:14:47,702 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 06:14:47,702 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 06:14:47,702 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:14:47,702 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:14:47,702 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20675.25 MB 2025-02-14 06:14:47,702 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21501.98 MB 2025-02-14 06:14:47,702 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 826.73 MB 2025-02-14 06:14:47,703 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23584.57 MB 2025-02-14 06:14:47,703 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23997.71 MB 2025-02-14 06:14:47,703 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 06:14:47,703 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22122.11 MB 2025-02-14 06:14:47,798 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 06:14:47,798 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 06:14:47,798 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 06:14:47,798 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:14:47,798 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21501.98 MB 2025-02-14 06:14:47,798 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22483.09 MB 2025-02-14 06:14:47,798 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 981.11 MB 2025-02-14 06:14:47,798 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23997.71 MB 2025-02-14 06:14:47,799 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26476.54 MB 2025-02-14 06:14:47,799 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2478.83 MB 2025-02-14 06:14:47,799 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24909.73 MB 2025-02-14 06:14:47,799 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 06:14:47,799 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 06:14:47,799 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 06:14:47,799 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:14:47,799 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20675.25 MB 2025-02-14 06:14:47,799 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22483.09 MB 2025-02-14 06:14:47,799 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1807.84 MB 2025-02-14 06:14:47,799 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23584.57 MB 2025-02-14 06:14:47,799 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26476.54 MB 2025-02-14 06:14:47,799 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2891.97 MB 2025-02-14 06:14:47,799 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24909.73 MB 2025-02-14 06:14:47,874 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 06:14:47,874 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 06:14:47,874 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 06:14:47,874 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:14:47,874 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23154.02 MB 2025-02-14 06:14:47,874 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23489.84 MB 2025-02-14 06:14:47,874 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 335.83 MB 2025-02-14 06:14:47,874 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26476.54 MB 2025-02-14 06:14:47,874 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26656.90 MB 2025-02-14 06:14:47,874 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 180.36 MB 2025-02-14 06:14:47,874 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23806.02 MB 2025-02-14 06:14:47,883 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 06:14:47,884 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 06:14:47,884 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:14:47,884 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:14:47,884 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23670.49 MB 2025-02-14 06:14:47,884 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23894.90 MB 2025-02-14 06:14:47,884 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 224.42 MB 2025-02-14 06:14:47,884 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26656.90 MB 2025-02-14 06:14:47,884 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26656.90 MB 2025-02-14 06:14:47,884 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:14:47,884 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23923.40 MB 2025-02-14 06:14:47,885 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 06:14:47,885 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 06:14:47,885 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.80 seconds 2025-02-14 06:14:47,885 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:14:47,885 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18913.99 MB 2025-02-14 06:14:47,885 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24095.78 MB 2025-02-14 06:14:47,885 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5181.79 MB 2025-02-14 06:14:47,885 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37868.27 MB 2025-02-14 06:14:47,885 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26656.90 MB 2025-02-14 06:14:47,885 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11211.37 MB 2025-02-14 06:14:47,885 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24095.78 MB 2025-02-14 06:14:48,153 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 06:14:48,153 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 06:14:48,153 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 06:14:48,153 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:14:48,153 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24095.78 MB 2025-02-14 06:14:48,153 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22853.58 MB 2025-02-14 06:14:48,153 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1242.20 MB 2025-02-14 06:14:48,153 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26656.90 MB 2025-02-14 06:14:48,153 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26656.90 MB 2025-02-14 06:14:48,153 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:14:48,153 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24329.97 MB 2025-02-14 06:14:48,171 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8154, cut from 8156 2025-02-14 06:14:48,171 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 06:14:48,177 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 06:14:48,177 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 06:14:48,177 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:14:48,177 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:14:48,177 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22853.58 MB 2025-02-14 06:14:48,177 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31284.26 MB 2025-02-14 06:14:48,177 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8430.68 MB 2025-02-14 06:14:48,177 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26656.90 MB 2025-02-14 06:14:48,177 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37136.37 MB 2025-02-14 06:14:48,177 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10479.47 MB 2025-02-14 06:14:48,177 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31284.26 MB 2025-02-14 06:14:48,339 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7946] 2025-02-14 06:14:48,341 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:14:48,341 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 06:14:48,342 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:14:48,342 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 06:14:48,346 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 06:14:48,347 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:14:48,347 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 06:14:48,347 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 06:16:14,289 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:16:14,289 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 06:16:14,297 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 06:16:14,303 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:16:14,303 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 184, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 06:16:14,305 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:16:14,305 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 184, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 06:16:17,207 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 06:16:17,208 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 06:16:17,208 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.90 seconds 2025-02-14 06:16:17,208 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:16:17,208 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19586.41 MB 2025-02-14 06:16:17,208 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20237.58 MB 2025-02-14 06:16:17,208 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 651.17 MB 2025-02-14 06:16:17,208 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49708.79 MB 2025-02-14 06:16:17,208 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22462.60 MB 2025-02-14 06:16:17,208 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27246.20 MB 2025-02-14 06:16:17,208 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29058.59 MB 2025-02-14 06:16:17,227 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 06:16:17,228 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 06:16:17,228 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:16:17,228 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:16:17,228 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20237.58 MB 2025-02-14 06:16:17,228 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20427.24 MB 2025-02-14 06:16:17,228 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 189.66 MB 2025-02-14 06:16:17,228 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22462.60 MB 2025-02-14 06:16:17,228 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23926.41 MB 2025-02-14 06:16:17,228 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1463.81 MB 2025-02-14 06:16:17,228 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22598.19 MB 2025-02-14 06:16:18,056 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 06:16:18,056 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 06:16:18,056 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.83 seconds 2025-02-14 06:16:18,057 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:16:18,057 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20427.24 MB 2025-02-14 06:16:18,057 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20647.54 MB 2025-02-14 06:16:18,057 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 220.30 MB 2025-02-14 06:16:18,057 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23926.41 MB 2025-02-14 06:16:18,057 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23163.04 MB 2025-02-14 06:16:18,057 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -763.36 MB 2025-02-14 06:16:18,057 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24598.72 MB 2025-02-14 06:16:18,068 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 06:16:18,068 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 06:16:18,068 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:16:18,068 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:16:18,068 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20647.47 MB 2025-02-14 06:16:18,068 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21431.97 MB 2025-02-14 06:16:18,068 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 784.49 MB 2025-02-14 06:16:18,068 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23163.04 MB 2025-02-14 06:16:18,068 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23555.21 MB 2025-02-14 06:16:18,068 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 392.17 MB 2025-02-14 06:16:18,068 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22020.20 MB 2025-02-14 06:16:18,189 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 06:16:18,189 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 06:16:18,189 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 06:16:18,189 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:16:18,190 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21431.97 MB 2025-02-14 06:16:18,190 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22362.90 MB 2025-02-14 06:16:18,190 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 930.93 MB 2025-02-14 06:16:18,190 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23555.21 MB 2025-02-14 06:16:18,190 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26105.35 MB 2025-02-14 06:16:18,190 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2550.14 MB 2025-02-14 06:16:18,190 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24665.83 MB 2025-02-14 06:16:18,191 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 06:16:18,191 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 06:16:18,191 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 06:16:18,191 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:16:18,191 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20647.47 MB 2025-02-14 06:16:18,191 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22362.90 MB 2025-02-14 06:16:18,191 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1715.42 MB 2025-02-14 06:16:18,191 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23163.04 MB 2025-02-14 06:16:18,191 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26105.35 MB 2025-02-14 06:16:18,191 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2942.30 MB 2025-02-14 06:16:18,191 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24665.83 MB 2025-02-14 06:16:18,307 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 06:16:18,307 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 06:16:18,307 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 06:16:18,307 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:16:18,307 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22999.32 MB 2025-02-14 06:16:18,307 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23317.62 MB 2025-02-14 06:16:18,307 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 318.31 MB 2025-02-14 06:16:18,307 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26105.35 MB 2025-02-14 06:16:18,307 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26275.22 MB 2025-02-14 06:16:18,307 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 169.87 MB 2025-02-14 06:16:18,308 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23619.01 MB 2025-02-14 06:16:18,323 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 06:16:18,323 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 06:16:18,323 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:16:18,323 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:16:18,323 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23488.98 MB 2025-02-14 06:16:18,323 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23707.95 MB 2025-02-14 06:16:18,323 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 218.97 MB 2025-02-14 06:16:18,323 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26275.22 MB 2025-02-14 06:16:18,323 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26275.22 MB 2025-02-14 06:16:18,323 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:16:18,323 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23728.10 MB 2025-02-14 06:16:18,325 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 06:16:18,325 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 06:16:18,325 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.02 seconds 2025-02-14 06:16:18,325 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:16:18,325 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18945.34 MB 2025-02-14 06:16:18,325 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23908.95 MB 2025-02-14 06:16:18,325 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4963.61 MB 2025-02-14 06:16:18,325 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49708.79 MB 2025-02-14 06:16:18,325 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26275.22 MB 2025-02-14 06:16:18,326 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23433.58 MB 2025-02-14 06:16:18,326 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23908.95 MB 2025-02-14 06:16:18,610 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 06:16:18,610 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 06:16:18,610 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.28 seconds 2025-02-14 06:16:18,611 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:16:18,611 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23908.95 MB 2025-02-14 06:16:18,611 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22844.01 MB 2025-02-14 06:16:18,611 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1064.94 MB 2025-02-14 06:16:18,611 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26275.22 MB 2025-02-14 06:16:18,611 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26275.22 MB 2025-02-14 06:16:18,611 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:16:18,611 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24511.53 MB 2025-02-14 06:16:18,642 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8159, cut from 8161 2025-02-14 06:16:18,642 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 06:16:18,649 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 06:16:18,649 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 06:16:18,649 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-14 06:16:18,649 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:16:18,649 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22844.01 MB 2025-02-14 06:16:18,650 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31279.61 MB 2025-02-14 06:16:18,650 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8435.59 MB 2025-02-14 06:16:18,650 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26275.22 MB 2025-02-14 06:16:18,650 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36760.98 MB 2025-02-14 06:16:18,650 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10485.76 MB 2025-02-14 06:16:18,650 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31279.61 MB 2025-02-14 06:16:18,909 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7951] 2025-02-14 06:16:18,912 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:16:18,912 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 06:16:18,914 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:16:18,914 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 06:16:18,921 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 06:16:18,923 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:16:18,924 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 06:16:18,924 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 06:16:28,415 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:16:28,415 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 06:16:28,420 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 06:16:28,424 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:16:28,424 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1853, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 06:16:28,425 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:16:28,425 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1853, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 06:16:57,012 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 06:16:57,012 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 06:16:57,012 - resource_logging.py:150 - __exit__ - DEBUG - Time: 28.58 seconds 2025-02-14 06:16:57,012 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:16:57,012 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31216.27 MB 2025-02-14 06:16:57,012 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37774.07 MB 2025-02-14 06:16:57,012 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6557.79 MB 2025-02-14 06:16:57,012 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45149.59 MB 2025-02-14 06:16:57,012 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45067.80 MB 2025-02-14 06:16:57,012 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -81.79 MB 2025-02-14 06:16:57,012 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46577.25 MB 2025-02-14 06:16:57,126 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 06:16:57,126 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 06:16:57,126 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 06:16:57,126 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:16:57,126 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37774.07 MB 2025-02-14 06:16:57,126 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30746.59 MB 2025-02-14 06:16:57,126 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7027.48 MB 2025-02-14 06:16:57,126 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45067.80 MB 2025-02-14 06:16:57,126 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 66230.16 MB 2025-02-14 06:16:57,126 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 21162.36 MB 2025-02-14 06:16:57,127 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 56791.28 MB 2025-02-14 06:16:59,067 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 06:16:59,067 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 06:16:59,067 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-14 06:16:59,067 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:16:59,067 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30746.59 MB 2025-02-14 06:16:59,067 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31277.43 MB 2025-02-14 06:16:59,067 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 06:16:59,067 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 66230.16 MB 2025-02-14 06:16:59,067 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35731.28 MB 2025-02-14 06:16:59,067 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30498.88 MB 2025-02-14 06:16:59,067 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35257.80 MB 2025-02-14 06:16:59,081 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 06:16:59,081 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 06:16:59,081 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:16:59,081 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:16:59,081 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31277.43 MB 2025-02-14 06:16:59,081 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33166.96 MB 2025-02-14 06:16:59,081 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 06:16:59,081 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35731.28 MB 2025-02-14 06:16:59,081 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37618.71 MB 2025-02-14 06:16:59,081 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 06:16:59,081 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34584.39 MB 2025-02-14 06:16:59,292 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 06:16:59,292 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 06:16:59,292 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 06:16:59,292 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:16:59,293 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33166.96 MB 2025-02-14 06:16:59,293 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35409.41 MB 2025-02-14 06:16:59,293 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2242.45 MB 2025-02-14 06:16:59,293 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37618.71 MB 2025-02-14 06:16:59,293 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43752.88 MB 2025-02-14 06:16:59,293 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 06:16:59,293 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40953.69 MB 2025-02-14 06:16:59,293 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 06:16:59,293 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 06:16:59,293 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 06:16:59,293 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:16:59,293 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31277.43 MB 2025-02-14 06:16:59,293 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35409.41 MB 2025-02-14 06:16:59,293 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.98 MB 2025-02-14 06:16:59,293 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35731.28 MB 2025-02-14 06:16:59,293 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43752.88 MB 2025-02-14 06:16:59,293 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-14 06:16:59,293 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40953.69 MB 2025-02-14 06:16:59,458 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 06:16:59,458 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 06:16:59,458 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 06:16:59,458 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:16:59,458 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36942.95 MB 2025-02-14 06:16:59,458 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37709.95 MB 2025-02-14 06:16:59,458 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 06:16:59,458 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43752.88 MB 2025-02-14 06:16:59,458 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44168.12 MB 2025-02-14 06:16:59,458 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 06:16:59,458 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38417.74 MB 2025-02-14 06:16:59,477 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 06:16:59,477 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 06:16:59,477 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:16:59,477 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:16:59,477 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38122.84 MB 2025-02-14 06:16:59,477 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38351.42 MB 2025-02-14 06:16:59,477 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.58 MB 2025-02-14 06:16:59,477 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44168.12 MB 2025-02-14 06:16:59,477 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44168.12 MB 2025-02-14 06:16:59,477 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:16:59,477 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38563.60 MB 2025-02-14 06:16:59,478 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 06:16:59,478 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 06:16:59,478 - resource_logging.py:150 - __exit__ - DEBUG - Time: 31.05 seconds 2025-02-14 06:16:59,478 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:16:59,478 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24760.27 MB 2025-02-14 06:16:59,478 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38551.73 MB 2025-02-14 06:16:59,478 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13791.46 MB 2025-02-14 06:16:59,478 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45149.59 MB 2025-02-14 06:16:59,478 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44168.12 MB 2025-02-14 06:16:59,478 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -981.47 MB 2025-02-14 06:16:59,478 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38563.60 MB 2025-02-14 06:16:59,747 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 06:16:59,747 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 06:16:59,747 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 06:16:59,747 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:16:59,747 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38551.73 MB 2025-02-14 06:16:59,747 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29753.05 MB 2025-02-14 06:16:59,747 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8798.68 MB 2025-02-14 06:16:59,747 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44168.12 MB 2025-02-14 06:16:59,747 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44168.12 MB 2025-02-14 06:16:59,747 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:16:59,747 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41053.87 MB 2025-02-14 06:16:59,765 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8131, cut from 8133 2025-02-14 06:16:59,765 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-14 06:16:59,771 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 06:16:59,771 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 06:16:59,771 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:16:59,771 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:16:59,771 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29753.05 MB 2025-02-14 06:16:59,771 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38160.79 MB 2025-02-14 06:16:59,771 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8407.74 MB 2025-02-14 06:16:59,771 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44168.12 MB 2025-02-14 06:16:59,771 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48347.74 MB 2025-02-14 06:16:59,771 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4179.62 MB 2025-02-14 06:16:59,771 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38160.79 MB 2025-02-14 06:16:59,932 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7923] 2025-02-14 06:16:59,933 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:16:59,933 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 06:16:59,934 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:16:59,934 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 06:16:59,939 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 06:16:59,940 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:16:59,940 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 06:16:59,940 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-14 06:18:06,213 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:18:06,213 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 06:18:06,218 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 06:18:06,222 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:18:06,222 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 171, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 06:18:06,223 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:18:06,223 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 171, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 06:18:08,870 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 06:18:08,870 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 06:18:08,870 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.64 seconds 2025-02-14 06:18:08,870 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:18:08,870 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19495.83 MB 2025-02-14 06:18:08,870 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20100.99 MB 2025-02-14 06:18:08,870 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 605.16 MB 2025-02-14 06:18:08,870 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56706.99 MB 2025-02-14 06:18:08,870 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22206.74 MB 2025-02-14 06:18:08,870 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34500.25 MB 2025-02-14 06:18:08,870 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28968.01 MB 2025-02-14 06:18:08,883 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 06:18:08,883 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 06:18:08,883 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:18:08,883 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:18:08,883 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20100.99 MB 2025-02-14 06:18:08,883 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20352.25 MB 2025-02-14 06:18:08,883 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 251.27 MB 2025-02-14 06:18:08,883 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22206.74 MB 2025-02-14 06:18:08,883 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23953.67 MB 2025-02-14 06:18:08,883 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1746.93 MB 2025-02-14 06:18:08,883 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22457.78 MB 2025-02-14 06:18:09,686 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 06:18:09,686 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 06:18:09,686 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.80 seconds 2025-02-14 06:18:09,686 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:18:09,686 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20352.25 MB 2025-02-14 06:18:09,686 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20571.22 MB 2025-02-14 06:18:09,686 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 218.97 MB 2025-02-14 06:18:09,686 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23953.67 MB 2025-02-14 06:18:09,686 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23368.56 MB 2025-02-14 06:18:09,686 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -585.11 MB 2025-02-14 06:18:09,686 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24523.73 MB 2025-02-14 06:18:09,694 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 06:18:09,694 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 06:18:09,694 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:18:09,694 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:18:09,694 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20571.16 MB 2025-02-14 06:18:09,694 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21351.19 MB 2025-02-14 06:18:09,694 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 780.03 MB 2025-02-14 06:18:09,694 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23368.56 MB 2025-02-14 06:18:09,694 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23758.64 MB 2025-02-14 06:18:09,694 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 390.07 MB 2025-02-14 06:18:09,694 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21935.88 MB 2025-02-14 06:18:09,784 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 06:18:09,784 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 06:18:09,784 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 06:18:09,785 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:18:09,785 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21351.19 MB 2025-02-14 06:18:09,785 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22276.78 MB 2025-02-14 06:18:09,785 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 925.59 MB 2025-02-14 06:18:09,785 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23758.64 MB 2025-02-14 06:18:09,785 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26294.09 MB 2025-02-14 06:18:09,785 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2535.46 MB 2025-02-14 06:18:09,785 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24567.30 MB 2025-02-14 06:18:09,785 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 06:18:09,785 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 06:18:09,785 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 06:18:09,785 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:18:09,785 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20571.16 MB 2025-02-14 06:18:09,785 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22276.78 MB 2025-02-14 06:18:09,785 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1705.62 MB 2025-02-14 06:18:09,785 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23368.56 MB 2025-02-14 06:18:09,785 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26294.09 MB 2025-02-14 06:18:09,785 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2925.53 MB 2025-02-14 06:18:09,785 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24567.30 MB 2025-02-14 06:18:09,852 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 06:18:09,852 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 06:18:09,852 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 06:18:09,852 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:18:09,852 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22909.37 MB 2025-02-14 06:18:09,852 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23226.15 MB 2025-02-14 06:18:09,852 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 316.78 MB 2025-02-14 06:18:09,852 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26294.09 MB 2025-02-14 06:18:09,852 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26461.86 MB 2025-02-14 06:18:09,852 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 167.77 MB 2025-02-14 06:18:09,852 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23527.08 MB 2025-02-14 06:18:09,861 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 06:18:09,862 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 06:18:09,862 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:18:09,862 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:18:09,862 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23396.47 MB 2025-02-14 06:18:09,862 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23612.75 MB 2025-02-14 06:18:09,862 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 216.28 MB 2025-02-14 06:18:09,862 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26461.86 MB 2025-02-14 06:18:09,862 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26461.86 MB 2025-02-14 06:18:09,862 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:18:09,862 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23626.47 MB 2025-02-14 06:18:09,863 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 06:18:09,863 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 06:18:09,863 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.64 seconds 2025-02-14 06:18:09,863 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:18:09,863 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18900.05 MB 2025-02-14 06:18:09,863 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23813.48 MB 2025-02-14 06:18:09,863 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4913.43 MB 2025-02-14 06:18:09,863 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56706.99 MB 2025-02-14 06:18:09,863 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26461.86 MB 2025-02-14 06:18:09,863 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30245.13 MB 2025-02-14 06:18:09,863 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23813.48 MB 2025-02-14 06:18:10,128 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 06:18:10,128 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 06:18:10,128 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 06:18:10,128 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:18:10,128 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23813.48 MB 2025-02-14 06:18:10,128 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22790.01 MB 2025-02-14 06:18:10,128 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1023.47 MB 2025-02-14 06:18:10,128 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26461.86 MB 2025-02-14 06:18:10,128 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26461.86 MB 2025-02-14 06:18:10,129 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:18:10,129 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24515.54 MB 2025-02-14 06:18:10,146 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8148, cut from 8150 2025-02-14 06:18:10,147 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 1,'] 2025-02-14 06:18:10,153 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 06:18:10,153 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 06:18:10,153 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:18:10,153 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:18:10,153 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22790.01 MB 2025-02-14 06:18:10,153 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31214.96 MB 2025-02-14 06:18:10,153 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8424.95 MB 2025-02-14 06:18:10,153 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26461.86 MB 2025-02-14 06:18:10,153 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36932.94 MB 2025-02-14 06:18:10,153 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10471.08 MB 2025-02-14 06:18:10,153 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31214.96 MB 2025-02-14 06:18:10,303 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7940] 2025-02-14 06:18:10,304 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:18:10,304 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 06:18:10,305 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:18:10,305 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 06:18:10,309 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 06:18:10,310 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:18:10,310 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 06:18:10,311 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 1,'] 2025-02-14 06:19:16,116 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:19:16,116 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 06:19:16,124 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 06:19:16,131 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:19:16,131 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1655, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 06:19:16,133 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:19:16,133 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1655, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 06:19:41,563 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 06:19:41,563 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 06:19:41,563 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.42 seconds 2025-02-14 06:19:41,563 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:19:41,563 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29836.58 MB 2025-02-14 06:19:41,563 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35693.92 MB 2025-02-14 06:19:41,563 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5857.35 MB 2025-02-14 06:19:41,563 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45308.97 MB 2025-02-14 06:19:41,563 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44342.18 MB 2025-02-14 06:19:41,563 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -966.79 MB 2025-02-14 06:19:41,563 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44518.08 MB 2025-02-14 06:19:41,652 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 06:19:41,652 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 06:19:41,653 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 06:19:41,653 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:19:41,653 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35693.92 MB 2025-02-14 06:19:41,653 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29717.25 MB 2025-02-14 06:19:41,653 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5976.68 MB 2025-02-14 06:19:41,653 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44342.18 MB 2025-02-14 06:19:41,653 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58598.62 MB 2025-02-14 06:19:41,653 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 14256.44 MB 2025-02-14 06:19:41,653 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50679.39 MB 2025-02-14 06:19:43,564 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 06:19:43,564 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 06:19:43,564 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 06:19:43,564 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:19:43,564 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29717.25 MB 2025-02-14 06:19:43,564 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30248.09 MB 2025-02-14 06:19:43,564 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 06:19:43,564 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58598.62 MB 2025-02-14 06:19:43,564 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35712.40 MB 2025-02-14 06:19:43,564 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22886.22 MB 2025-02-14 06:19:43,564 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34227.42 MB 2025-02-14 06:19:43,580 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 06:19:43,580 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 06:19:43,580 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:19:43,580 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:19:43,580 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30248.09 MB 2025-02-14 06:19:43,580 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32137.62 MB 2025-02-14 06:19:43,580 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 06:19:43,581 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35712.40 MB 2025-02-14 06:19:43,581 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37599.84 MB 2025-02-14 06:19:43,581 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 06:19:43,581 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33555.05 MB 2025-02-14 06:19:43,783 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 06:19:43,783 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 06:19:43,783 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 06:19:43,783 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:19:43,783 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32137.62 MB 2025-02-14 06:19:43,783 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34380.07 MB 2025-02-14 06:19:43,783 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2242.45 MB 2025-02-14 06:19:43,783 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37599.84 MB 2025-02-14 06:19:43,783 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43262.15 MB 2025-02-14 06:19:43,783 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 06:19:43,783 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39924.35 MB 2025-02-14 06:19:43,784 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 06:19:43,784 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 06:19:43,784 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 06:19:43,784 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:19:43,784 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30248.09 MB 2025-02-14 06:19:43,784 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34380.07 MB 2025-02-14 06:19:43,784 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.98 MB 2025-02-14 06:19:43,784 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35712.40 MB 2025-02-14 06:19:43,784 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43262.15 MB 2025-02-14 06:19:43,784 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-14 06:19:43,784 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39924.35 MB 2025-02-14 06:19:43,942 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 06:19:43,942 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 06:19:43,942 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 06:19:43,942 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:19:43,942 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35913.61 MB 2025-02-14 06:19:43,942 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36680.61 MB 2025-02-14 06:19:43,942 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 06:19:43,942 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43262.15 MB 2025-02-14 06:19:43,942 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43677.38 MB 2025-02-14 06:19:43,942 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 06:19:43,942 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37388.40 MB 2025-02-14 06:19:43,960 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 06:19:43,960 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 06:19:43,960 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:19:43,960 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:19:43,960 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37093.50 MB 2025-02-14 06:19:43,960 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37321.85 MB 2025-02-14 06:19:43,960 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.35 MB 2025-02-14 06:19:43,960 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43677.38 MB 2025-02-14 06:19:43,960 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43677.38 MB 2025-02-14 06:19:43,960 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:19:43,960 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37531.21 MB 2025-02-14 06:19:43,961 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 06:19:43,961 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 06:19:43,961 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.82 seconds 2025-02-14 06:19:43,961 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:19:43,961 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24070.42 MB 2025-02-14 06:19:43,961 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37522.11 MB 2025-02-14 06:19:43,961 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13451.68 MB 2025-02-14 06:19:43,961 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45308.97 MB 2025-02-14 06:19:43,961 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43677.38 MB 2025-02-14 06:19:43,961 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1631.58 MB 2025-02-14 06:19:43,961 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37531.21 MB 2025-02-14 06:19:44,228 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 06:19:44,228 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 06:19:44,228 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 06:19:44,229 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:19:44,229 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37522.11 MB 2025-02-14 06:19:44,229 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29062.46 MB 2025-02-14 06:19:44,229 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8459.64 MB 2025-02-14 06:19:44,229 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43677.38 MB 2025-02-14 06:19:44,229 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43677.38 MB 2025-02-14 06:19:44,229 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:19:44,229 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40023.64 MB 2025-02-14 06:19:44,246 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8129, cut from 8131 2025-02-14 06:19:44,247 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 06:19:44,253 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 06:19:44,253 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 06:19:44,253 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:19:44,253 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:19:44,253 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29062.46 MB 2025-02-14 06:19:44,253 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37467.58 MB 2025-02-14 06:19:44,253 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8405.11 MB 2025-02-14 06:19:44,253 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43677.38 MB 2025-02-14 06:19:44,253 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52034.54 MB 2025-02-14 06:19:44,253 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8357.15 MB 2025-02-14 06:19:44,253 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37467.58 MB 2025-02-14 06:19:44,406 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7921] 2025-02-14 06:19:44,408 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:19:44,408 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 06:19:44,409 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:19:44,409 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 06:19:44,413 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 06:19:44,414 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:19:44,414 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 06:19:44,414 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 06:21:47,480 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:21:47,481 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 06:21:47,486 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 06:21:47,491 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:21:47,491 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1353, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 06:21:47,492 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:21:47,492 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1353, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 06:22:08,204 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 06:22:08,204 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 06:22:08,204 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.70 seconds 2025-02-14 06:22:08,204 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:22:08,204 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27732.19 MB 2025-02-14 06:22:08,204 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32520.38 MB 2025-02-14 06:22:08,204 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4788.19 MB 2025-02-14 06:22:08,204 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64569.21 MB 2025-02-14 06:22:08,204 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43243.27 MB 2025-02-14 06:22:08,204 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21325.94 MB 2025-02-14 06:22:08,204 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41506.92 MB 2025-02-14 06:22:08,282 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 06:22:08,282 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 06:22:08,282 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 06:22:08,282 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:22:08,282 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32520.38 MB 2025-02-14 06:22:08,282 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28147.24 MB 2025-02-14 06:22:08,282 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4373.14 MB 2025-02-14 06:22:08,282 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43243.27 MB 2025-02-14 06:22:08,282 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52374.27 MB 2025-02-14 06:22:08,282 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9131.00 MB 2025-02-14 06:22:08,282 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46284.10 MB 2025-02-14 06:22:10,202 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 06:22:10,202 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 06:22:10,202 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 06:22:10,202 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:22:10,202 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28147.24 MB 2025-02-14 06:22:10,202 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28678.08 MB 2025-02-14 06:22:10,202 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 06:22:10,202 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52374.27 MB 2025-02-14 06:22:10,202 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34275.85 MB 2025-02-14 06:22:10,202 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18098.42 MB 2025-02-14 06:22:10,202 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32657.42 MB 2025-02-14 06:22:10,219 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 06:22:10,219 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 06:22:10,219 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:22:10,219 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:22:10,219 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28678.08 MB 2025-02-14 06:22:10,219 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30567.62 MB 2025-02-14 06:22:10,219 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 06:22:10,219 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34275.85 MB 2025-02-14 06:22:10,219 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36163.29 MB 2025-02-14 06:22:10,219 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 06:22:10,219 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31985.05 MB 2025-02-14 06:22:10,434 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 06:22:10,435 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 06:22:10,435 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 06:22:10,435 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:22:10,435 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30567.62 MB 2025-02-14 06:22:10,435 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32810.06 MB 2025-02-14 06:22:10,435 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2242.45 MB 2025-02-14 06:22:10,435 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36163.29 MB 2025-02-14 06:22:10,435 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41825.60 MB 2025-02-14 06:22:10,435 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 06:22:10,435 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38354.34 MB 2025-02-14 06:22:10,435 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 06:22:10,435 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 06:22:10,435 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 06:22:10,435 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:22:10,435 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28678.08 MB 2025-02-14 06:22:10,435 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32810.06 MB 2025-02-14 06:22:10,435 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.98 MB 2025-02-14 06:22:10,435 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34275.85 MB 2025-02-14 06:22:10,436 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41825.60 MB 2025-02-14 06:22:10,436 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-14 06:22:10,436 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38354.34 MB 2025-02-14 06:22:10,605 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 06:22:10,605 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 06:22:10,605 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 06:22:10,605 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:22:10,605 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34343.61 MB 2025-02-14 06:22:10,605 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35110.61 MB 2025-02-14 06:22:10,605 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 06:22:10,605 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41825.60 MB 2025-02-14 06:22:10,605 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42240.84 MB 2025-02-14 06:22:10,605 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 06:22:10,605 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35818.40 MB 2025-02-14 06:22:10,624 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 06:22:10,624 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 06:22:10,624 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:22:10,624 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:22:10,624 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35523.50 MB 2025-02-14 06:22:10,624 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35753.40 MB 2025-02-14 06:22:10,624 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.91 MB 2025-02-14 06:22:10,624 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42240.84 MB 2025-02-14 06:22:10,624 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42240.84 MB 2025-02-14 06:22:10,624 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:22:10,624 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35958.09 MB 2025-02-14 06:22:10,625 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 06:22:10,625 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 06:22:10,625 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.13 seconds 2025-02-14 06:22:10,625 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:22:10,625 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23018.23 MB 2025-02-14 06:22:10,625 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35954.48 MB 2025-02-14 06:22:10,625 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12936.24 MB 2025-02-14 06:22:10,625 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64569.21 MB 2025-02-14 06:22:10,625 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42240.84 MB 2025-02-14 06:22:10,625 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22328.38 MB 2025-02-14 06:22:10,625 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35958.09 MB 2025-02-14 06:22:10,892 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 06:22:10,892 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 06:22:10,892 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 06:22:10,893 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:22:10,893 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35954.48 MB 2025-02-14 06:22:10,893 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28022.62 MB 2025-02-14 06:22:10,893 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7931.85 MB 2025-02-14 06:22:10,893 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42240.84 MB 2025-02-14 06:22:10,893 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42240.84 MB 2025-02-14 06:22:10,893 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:22:10,893 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38466.14 MB 2025-02-14 06:22:10,910 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 06:22:10,911 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-14 06:22:10,917 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 06:22:10,917 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 06:22:10,917 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:22:10,917 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:22:10,917 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28022.62 MB 2025-02-14 06:22:10,917 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36461.64 MB 2025-02-14 06:22:10,917 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 06:22:10,917 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42240.84 MB 2025-02-14 06:22:10,917 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50631.54 MB 2025-02-14 06:22:10,917 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 06:22:10,917 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36461.64 MB 2025-02-14 06:22:11,080 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 06:22:11,081 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:22:11,081 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 06:22:11,082 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:22:11,082 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 06:22:11,087 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 06:22:11,088 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:22:11,088 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 06:22:11,088 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-14 06:22:59,247 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:22:59,247 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 06:22:59,252 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 06:22:59,258 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:22:59,258 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2619, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 06:22:59,260 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:22:59,260 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2619, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 06:23:39,956 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 06:23:39,956 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 06:23:39,957 - resource_logging.py:150 - __exit__ - DEBUG - Time: 40.68 seconds 2025-02-14 06:23:39,957 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:23:39,957 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36554.80 MB 2025-02-14 06:23:39,957 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45824.21 MB 2025-02-14 06:23:39,957 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 9269.41 MB 2025-02-14 06:23:39,957 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 81470.16 MB 2025-02-14 06:23:39,957 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49255.81 MB 2025-02-14 06:23:39,957 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32214.35 MB 2025-02-14 06:23:39,957 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 55092.71 MB 2025-02-14 06:23:40,181 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 06:23:40,181 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 06:23:40,181 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 06:23:40,181 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:23:40,181 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45824.21 MB 2025-02-14 06:23:40,181 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34730.29 MB 2025-02-14 06:23:40,181 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -11093.92 MB 2025-02-14 06:23:40,181 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49255.81 MB 2025-02-14 06:23:40,181 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 85658.17 MB 2025-02-14 06:23:40,181 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 36402.36 MB 2025-02-14 06:23:40,181 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 72674.63 MB 2025-02-14 06:23:42,156 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 06:23:42,156 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 06:23:42,156 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.97 seconds 2025-02-14 06:23:42,157 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:23:42,157 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34730.29 MB 2025-02-14 06:23:42,157 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35261.13 MB 2025-02-14 06:23:42,157 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 06:23:42,157 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 85658.17 MB 2025-02-14 06:23:42,157 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37476.11 MB 2025-02-14 06:23:42,157 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -48182.07 MB 2025-02-14 06:23:42,157 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39241.50 MB 2025-02-14 06:23:42,171 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 06:23:42,171 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 06:23:42,171 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:23:42,171 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:23:42,171 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35261.13 MB 2025-02-14 06:23:42,171 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37150.66 MB 2025-02-14 06:23:42,171 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 06:23:42,171 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37476.11 MB 2025-02-14 06:23:42,171 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41250.98 MB 2025-02-14 06:23:42,171 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 06:23:42,171 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38568.09 MB 2025-02-14 06:23:42,383 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 06:23:42,383 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 06:23:42,383 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 06:23:42,383 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:23:42,383 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37150.66 MB 2025-02-14 06:23:42,383 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39393.11 MB 2025-02-14 06:23:42,383 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2242.45 MB 2025-02-14 06:23:42,383 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41250.98 MB 2025-02-14 06:23:42,383 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47385.15 MB 2025-02-14 06:23:42,383 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 06:23:42,383 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44937.39 MB 2025-02-14 06:23:42,384 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 06:23:42,384 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 06:23:42,384 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 06:23:42,384 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:23:42,384 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35261.13 MB 2025-02-14 06:23:42,384 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39393.11 MB 2025-02-14 06:23:42,384 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.98 MB 2025-02-14 06:23:42,384 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37476.11 MB 2025-02-14 06:23:42,384 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47385.15 MB 2025-02-14 06:23:42,384 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9909.04 MB 2025-02-14 06:23:42,384 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44937.39 MB 2025-02-14 06:23:42,552 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 06:23:42,552 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 06:23:42,552 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 06:23:42,552 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:23:42,552 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40926.65 MB 2025-02-14 06:23:42,552 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41693.65 MB 2025-02-14 06:23:42,552 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 06:23:42,552 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47385.15 MB 2025-02-14 06:23:42,552 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47800.39 MB 2025-02-14 06:23:42,553 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 06:23:42,553 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42401.44 MB 2025-02-14 06:23:42,572 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 06:23:42,572 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 06:23:42,572 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:23:42,572 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:23:42,572 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42106.54 MB 2025-02-14 06:23:42,572 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42334.33 MB 2025-02-14 06:23:42,572 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.79 MB 2025-02-14 06:23:42,572 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47800.39 MB 2025-02-14 06:23:42,572 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47800.39 MB 2025-02-14 06:23:42,572 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:23:42,572 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42551.21 MB 2025-02-14 06:23:42,573 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 06:23:42,573 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 06:23:42,573 - resource_logging.py:150 - __exit__ - DEBUG - Time: 43.31 seconds 2025-02-14 06:23:42,573 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:23:42,573 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27429.54 MB 2025-02-14 06:23:42,573 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42534.59 MB 2025-02-14 06:23:42,573 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 15105.06 MB 2025-02-14 06:23:42,573 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 72343.36 MB 2025-02-14 06:23:42,573 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47800.39 MB 2025-02-14 06:23:42,573 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24542.97 MB 2025-02-14 06:23:42,573 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42551.21 MB 2025-02-14 06:23:42,846 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 06:23:42,846 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 06:23:42,846 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 06:23:42,846 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:23:42,846 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42534.59 MB 2025-02-14 06:23:42,846 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32421.02 MB 2025-02-14 06:23:42,846 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10113.57 MB 2025-02-14 06:23:42,846 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47800.39 MB 2025-02-14 06:23:42,846 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47800.39 MB 2025-02-14 06:23:42,846 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:23:42,846 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45036.12 MB 2025-02-14 06:23:42,865 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8129, cut from 8131 2025-02-14 06:23:42,865 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-14 06:23:42,871 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 06:23:42,871 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 06:23:42,871 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:23:42,871 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:23:42,871 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32421.02 MB 2025-02-14 06:23:42,871 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40825.65 MB 2025-02-14 06:23:42,871 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8404.63 MB 2025-02-14 06:23:42,871 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47800.39 MB 2025-02-14 06:23:42,871 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51980.01 MB 2025-02-14 06:23:42,871 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4179.62 MB 2025-02-14 06:23:42,871 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40825.65 MB 2025-02-14 06:23:43,033 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7921] 2025-02-14 06:23:43,035 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:23:43,035 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 06:23:43,036 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:23:43,036 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 06:23:43,040 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 06:23:43,041 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:23:43,041 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 06:23:43,042 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-14 06:24:36,530 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:24:36,530 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 06:24:36,535 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 06:24:36,539 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:24:36,539 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1201, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 06:24:36,540 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:24:36,540 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1201, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 06:24:55,203 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 06:24:55,204 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 06:24:55,204 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.66 seconds 2025-02-14 06:24:55,204 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:24:55,204 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26673.03 MB 2025-02-14 06:24:55,204 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30923.96 MB 2025-02-14 06:24:55,204 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4250.93 MB 2025-02-14 06:24:55,204 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64514.69 MB 2025-02-14 06:24:55,204 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34653.34 MB 2025-02-14 06:24:55,204 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29861.35 MB 2025-02-14 06:24:55,204 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39769.09 MB 2025-02-14 06:24:55,278 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 06:24:55,278 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 06:24:55,278 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 06:24:55,278 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:24:55,278 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30923.96 MB 2025-02-14 06:24:55,278 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27358.09 MB 2025-02-14 06:24:55,278 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3565.87 MB 2025-02-14 06:24:55,278 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34653.34 MB 2025-02-14 06:24:55,279 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46286.24 MB 2025-02-14 06:24:55,279 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 11632.90 MB 2025-02-14 06:24:55,279 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40927.18 MB 2025-02-14 06:24:57,211 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 06:24:57,211 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 06:24:57,211 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 06:24:57,211 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:24:57,211 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27358.09 MB 2025-02-14 06:24:57,211 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27888.93 MB 2025-02-14 06:24:57,211 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 06:24:57,211 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46286.24 MB 2025-02-14 06:24:57,211 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32526.83 MB 2025-02-14 06:24:57,211 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13759.41 MB 2025-02-14 06:24:57,211 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31868.26 MB 2025-02-14 06:24:57,225 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 06:24:57,225 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 06:24:57,225 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:24:57,225 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:24:57,225 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27888.93 MB 2025-02-14 06:24:57,225 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29778.47 MB 2025-02-14 06:24:57,225 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 06:24:57,225 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32526.83 MB 2025-02-14 06:24:57,225 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34414.26 MB 2025-02-14 06:24:57,225 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 06:24:57,225 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31195.89 MB 2025-02-14 06:24:57,440 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 06:24:57,440 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 06:24:57,440 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 06:24:57,440 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:24:57,440 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29778.47 MB 2025-02-14 06:24:57,440 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32020.91 MB 2025-02-14 06:24:57,440 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2242.45 MB 2025-02-14 06:24:57,440 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34414.26 MB 2025-02-14 06:24:57,440 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40076.57 MB 2025-02-14 06:24:57,440 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 06:24:57,440 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37565.19 MB 2025-02-14 06:24:57,441 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 06:24:57,441 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 06:24:57,441 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 06:24:57,441 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:24:57,441 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27888.93 MB 2025-02-14 06:24:57,441 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32020.91 MB 2025-02-14 06:24:57,441 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.98 MB 2025-02-14 06:24:57,441 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32526.83 MB 2025-02-14 06:24:57,441 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40076.57 MB 2025-02-14 06:24:57,441 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-14 06:24:57,441 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37565.19 MB 2025-02-14 06:24:57,615 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 06:24:57,615 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 06:24:57,615 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 06:24:57,615 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:24:57,615 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33554.45 MB 2025-02-14 06:24:57,615 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34321.46 MB 2025-02-14 06:24:57,615 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 06:24:57,615 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40076.57 MB 2025-02-14 06:24:57,615 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40491.81 MB 2025-02-14 06:24:57,615 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 06:24:57,615 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35029.24 MB 2025-02-14 06:24:57,634 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 06:24:57,634 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 06:24:57,634 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:24:57,634 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:24:57,634 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34734.34 MB 2025-02-14 06:24:57,634 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34965.12 MB 2025-02-14 06:24:57,634 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 230.78 MB 2025-02-14 06:24:57,634 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40491.81 MB 2025-02-14 06:24:57,634 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40491.81 MB 2025-02-14 06:24:57,634 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:24:57,634 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35189.31 MB 2025-02-14 06:24:57,635 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 06:24:57,635 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 06:24:57,635 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.09 seconds 2025-02-14 06:24:57,635 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:24:57,635 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22488.65 MB 2025-02-14 06:24:57,636 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35166.20 MB 2025-02-14 06:24:57,636 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12677.54 MB 2025-02-14 06:24:57,636 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64514.69 MB 2025-02-14 06:24:57,636 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40491.81 MB 2025-02-14 06:24:57,636 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24022.88 MB 2025-02-14 06:24:57,636 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35189.31 MB 2025-02-14 06:24:57,905 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 06:24:57,905 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 06:24:57,905 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 06:24:57,905 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:24:57,905 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35166.20 MB 2025-02-14 06:24:57,905 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27493.04 MB 2025-02-14 06:24:57,905 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7673.16 MB 2025-02-14 06:24:57,905 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40491.81 MB 2025-02-14 06:24:57,905 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40491.81 MB 2025-02-14 06:24:57,905 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:24:57,905 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37677.86 MB 2025-02-14 06:24:57,923 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 06:24:57,924 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 06:24:57,930 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 06:24:57,930 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 06:24:57,930 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:24:57,930 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:24:57,930 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27493.04 MB 2025-02-14 06:24:57,930 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35932.06 MB 2025-02-14 06:24:57,930 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 06:24:57,930 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40491.81 MB 2025-02-14 06:24:57,930 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48882.52 MB 2025-02-14 06:24:57,930 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 06:24:57,930 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35932.06 MB 2025-02-14 06:24:58,099 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 06:24:58,101 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:24:58,101 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 06:24:58,102 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:24:58,102 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 06:24:58,107 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 06:24:58,108 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:24:58,108 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 06:24:58,108 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 06:25:27,953 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:25:27,954 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 06:25:27,959 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 06:25:27,962 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:25:27,962 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1254, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 06:25:27,963 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:25:27,963 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1254, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 06:25:47,469 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 06:25:47,470 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 06:25:47,470 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.50 seconds 2025-02-14 06:25:47,470 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:25:47,470 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27042.35 MB 2025-02-14 06:25:47,470 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31480.18 MB 2025-02-14 06:25:47,470 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4437.84 MB 2025-02-14 06:25:47,470 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61467.53 MB 2025-02-14 06:25:47,470 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42932.90 MB 2025-02-14 06:25:47,470 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18534.63 MB 2025-02-14 06:25:47,470 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40364.09 MB 2025-02-14 06:25:47,546 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 06:25:47,546 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 06:25:47,546 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 06:25:47,546 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:25:47,546 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31480.18 MB 2025-02-14 06:25:47,546 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27632.57 MB 2025-02-14 06:25:47,546 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3847.61 MB 2025-02-14 06:25:47,546 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42932.90 MB 2025-02-14 06:25:47,546 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49218.06 MB 2025-02-14 06:25:47,546 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6285.16 MB 2025-02-14 06:25:47,546 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44041.46 MB 2025-02-14 06:25:49,475 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 06:25:49,475 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 06:25:49,475 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 06:25:49,475 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:25:49,475 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27632.57 MB 2025-02-14 06:25:49,475 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28163.41 MB 2025-02-14 06:25:49,475 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 06:25:49,475 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49218.06 MB 2025-02-14 06:25:49,475 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34315.70 MB 2025-02-14 06:25:49,475 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14902.36 MB 2025-02-14 06:25:49,475 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32142.96 MB 2025-02-14 06:25:49,491 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 06:25:49,491 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 06:25:49,491 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:25:49,491 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:25:49,491 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28163.41 MB 2025-02-14 06:25:49,491 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30052.95 MB 2025-02-14 06:25:49,491 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 06:25:49,491 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34315.70 MB 2025-02-14 06:25:49,491 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35259.42 MB 2025-02-14 06:25:49,491 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 06:25:49,491 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31470.38 MB 2025-02-14 06:25:49,701 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 06:25:49,701 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 06:25:49,701 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 06:25:49,702 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:25:49,702 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30052.95 MB 2025-02-14 06:25:49,702 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32295.39 MB 2025-02-14 06:25:49,702 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2242.45 MB 2025-02-14 06:25:49,702 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35259.42 MB 2025-02-14 06:25:49,702 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40921.73 MB 2025-02-14 06:25:49,702 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 06:25:49,702 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37839.67 MB 2025-02-14 06:25:49,702 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 06:25:49,702 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 06:25:49,702 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 06:25:49,702 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:25:49,702 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28163.41 MB 2025-02-14 06:25:49,702 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32295.39 MB 2025-02-14 06:25:49,702 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.98 MB 2025-02-14 06:25:49,702 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34315.70 MB 2025-02-14 06:25:49,702 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40921.73 MB 2025-02-14 06:25:49,702 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 06:25:49,702 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37839.67 MB 2025-02-14 06:25:49,869 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 06:25:49,869 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 06:25:49,869 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 06:25:49,869 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:25:49,869 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33828.94 MB 2025-02-14 06:25:49,869 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34595.94 MB 2025-02-14 06:25:49,869 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 06:25:49,869 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40921.73 MB 2025-02-14 06:25:49,869 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41336.96 MB 2025-02-14 06:25:49,869 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 06:25:49,869 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35303.73 MB 2025-02-14 06:25:49,887 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 06:25:49,887 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 06:25:49,887 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:25:49,887 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:25:49,887 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35008.83 MB 2025-02-14 06:25:49,887 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35236.19 MB 2025-02-14 06:25:49,887 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.37 MB 2025-02-14 06:25:49,887 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41336.96 MB 2025-02-14 06:25:49,887 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41336.96 MB 2025-02-14 06:25:49,887 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:25:49,888 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35449.52 MB 2025-02-14 06:25:49,889 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 06:25:49,889 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 06:25:49,889 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.92 seconds 2025-02-14 06:25:49,889 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:25:49,889 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22673.31 MB 2025-02-14 06:25:49,889 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35437.27 MB 2025-02-14 06:25:49,889 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12763.96 MB 2025-02-14 06:25:49,889 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61467.53 MB 2025-02-14 06:25:49,889 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41336.96 MB 2025-02-14 06:25:49,889 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20130.56 MB 2025-02-14 06:25:49,889 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35449.52 MB 2025-02-14 06:25:50,158 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 06:25:50,158 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 06:25:50,158 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 06:25:50,158 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:25:50,158 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35437.27 MB 2025-02-14 06:25:50,158 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27677.70 MB 2025-02-14 06:25:50,158 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7759.57 MB 2025-02-14 06:25:50,158 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41336.96 MB 2025-02-14 06:25:50,158 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41336.96 MB 2025-02-14 06:25:50,158 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:25:50,158 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37948.93 MB 2025-02-14 06:25:50,176 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 06:25:50,176 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 06:25:50,182 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 06:25:50,182 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 06:25:50,182 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:25:50,182 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:25:50,182 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27677.70 MB 2025-02-14 06:25:50,182 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36116.72 MB 2025-02-14 06:25:50,182 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 06:25:50,182 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41336.96 MB 2025-02-14 06:25:50,182 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49727.67 MB 2025-02-14 06:25:50,182 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 06:25:50,182 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36116.72 MB 2025-02-14 06:25:50,347 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 06:25:50,349 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:25:50,349 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 06:25:50,350 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:25:50,350 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 06:25:50,355 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 06:25:50,356 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:25:50,356 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 06:25:50,356 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 06:26:06,265 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:26:06,265 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 06:26:06,270 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 06:26:06,273 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:26:06,273 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 529, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 06:26:06,274 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:26:06,274 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 529, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 06:26:14,625 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 06:26:14,626 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 06:26:14,626 - resource_logging.py:150 - __exit__ - DEBUG - Time: 8.35 seconds 2025-02-14 06:26:14,626 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:26:14,626 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21990.43 MB 2025-02-14 06:26:14,626 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23863.19 MB 2025-02-14 06:26:14,626 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1872.76 MB 2025-02-14 06:26:14,626 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62312.68 MB 2025-02-14 06:26:14,626 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27380.42 MB 2025-02-14 06:26:14,626 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34932.26 MB 2025-02-14 06:26:14,626 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32821.56 MB 2025-02-14 06:26:14,664 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 06:26:14,665 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 06:26:14,665 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-14 06:26:14,665 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:26:14,665 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23863.19 MB 2025-02-14 06:26:14,665 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23864.57 MB 2025-02-14 06:26:14,665 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1.39 MB 2025-02-14 06:26:14,665 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27380.42 MB 2025-02-14 06:26:14,665 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34928.07 MB 2025-02-14 06:26:14,665 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7547.65 MB 2025-02-14 06:26:14,665 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31696.55 MB 2025-02-14 06:26:16,610 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 06:26:16,610 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 06:26:16,610 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-14 06:26:16,610 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:26:16,610 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23864.57 MB 2025-02-14 06:26:16,610 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24395.41 MB 2025-02-14 06:26:16,610 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 06:26:16,610 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34928.07 MB 2025-02-14 06:26:16,610 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26688.36 MB 2025-02-14 06:26:16,610 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8239.71 MB 2025-02-14 06:26:16,610 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28375.78 MB 2025-02-14 06:26:16,624 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 06:26:16,624 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 06:26:16,624 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:26:16,624 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:26:16,624 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24395.41 MB 2025-02-14 06:26:16,624 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26284.95 MB 2025-02-14 06:26:16,624 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 06:26:16,624 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26688.36 MB 2025-02-14 06:26:16,624 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30463.23 MB 2025-02-14 06:26:16,624 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 06:26:16,624 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27702.38 MB 2025-02-14 06:26:16,834 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 06:26:16,834 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 06:26:16,834 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 06:26:16,834 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:26:16,834 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26284.95 MB 2025-02-14 06:26:16,834 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28527.39 MB 2025-02-14 06:26:16,834 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2242.45 MB 2025-02-14 06:26:16,834 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30463.23 MB 2025-02-14 06:26:16,834 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36597.40 MB 2025-02-14 06:26:16,834 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 06:26:16,835 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34071.67 MB 2025-02-14 06:26:16,835 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 06:26:16,835 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 06:26:16,835 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 06:26:16,835 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:26:16,835 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24395.41 MB 2025-02-14 06:26:16,835 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28527.39 MB 2025-02-14 06:26:16,835 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.98 MB 2025-02-14 06:26:16,835 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26688.36 MB 2025-02-14 06:26:16,835 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36597.40 MB 2025-02-14 06:26:16,835 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9909.04 MB 2025-02-14 06:26:16,835 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34071.67 MB 2025-02-14 06:26:17,003 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 06:26:17,003 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 06:26:17,003 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 06:26:17,003 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:26:17,003 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30060.94 MB 2025-02-14 06:26:17,003 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30827.94 MB 2025-02-14 06:26:17,003 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 06:26:17,003 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36597.40 MB 2025-02-14 06:26:17,003 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37012.64 MB 2025-02-14 06:26:17,003 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 06:26:17,003 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31535.73 MB 2025-02-14 06:26:17,022 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 06:26:17,022 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 06:26:17,022 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:26:17,022 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:26:17,022 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31240.83 MB 2025-02-14 06:26:17,022 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31470.39 MB 2025-02-14 06:26:17,022 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.56 MB 2025-02-14 06:26:17,022 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37012.64 MB 2025-02-14 06:26:17,022 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37012.64 MB 2025-02-14 06:26:17,022 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:26:17,022 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31692.23 MB 2025-02-14 06:26:17,023 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 06:26:17,023 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 06:26:17,023 - resource_logging.py:150 - __exit__ - DEBUG - Time: 10.75 seconds 2025-02-14 06:26:17,023 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:26:17,023 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20147.35 MB 2025-02-14 06:26:17,023 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31671.46 MB 2025-02-14 06:26:17,023 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11524.11 MB 2025-02-14 06:26:17,023 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62312.68 MB 2025-02-14 06:26:17,023 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37012.64 MB 2025-02-14 06:26:17,023 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25300.04 MB 2025-02-14 06:26:17,023 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31692.23 MB 2025-02-14 06:26:17,295 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 06:26:17,295 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 06:26:17,295 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 06:26:17,295 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:26:17,295 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31671.46 MB 2025-02-14 06:26:17,295 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25151.74 MB 2025-02-14 06:26:17,295 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6519.72 MB 2025-02-14 06:26:17,295 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37012.64 MB 2025-02-14 06:26:17,295 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37012.64 MB 2025-02-14 06:26:17,295 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:26:17,295 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34183.13 MB 2025-02-14 06:26:17,317 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 06:26:17,317 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-14 06:26:17,323 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 06:26:17,323 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 06:26:17,323 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 06:26:17,323 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:26:17,323 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25151.74 MB 2025-02-14 06:26:17,323 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33590.76 MB 2025-02-14 06:26:17,323 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 06:26:17,323 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37012.64 MB 2025-02-14 06:26:17,323 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47502.59 MB 2025-02-14 06:26:17,323 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 06:26:17,323 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33590.76 MB 2025-02-14 06:26:17,490 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 06:26:17,491 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:26:17,491 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 06:26:17,492 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:26:17,492 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 06:26:17,497 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 06:26:17,498 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:26:17,498 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 06:26:17,498 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-14 06:27:48,666 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:27:48,666 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 06:27:48,671 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 06:27:48,675 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:27:48,675 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 301, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 06:27:48,676 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:27:48,676 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 301, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 06:27:53,325 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 06:27:53,326 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 06:27:53,326 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.64 seconds 2025-02-14 06:27:53,326 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:27:53,326 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20401.69 MB 2025-02-14 06:27:53,326 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21467.04 MB 2025-02-14 06:27:53,326 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1065.35 MB 2025-02-14 06:27:53,326 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60087.60 MB 2025-02-14 06:27:53,326 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24452.79 MB 2025-02-14 06:27:53,326 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35634.81 MB 2025-02-14 06:27:53,326 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30326.85 MB 2025-02-14 06:27:53,347 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 06:27:53,347 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 06:27:53,347 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:27:53,347 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:27:53,347 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21467.04 MB 2025-02-14 06:27:53,348 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21983.14 MB 2025-02-14 06:27:53,348 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 516.10 MB 2025-02-14 06:27:53,348 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24452.79 MB 2025-02-14 06:27:53,348 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28684.85 MB 2025-02-14 06:27:53,348 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4232.05 MB 2025-02-14 06:27:53,348 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25741.30 MB 2025-02-14 06:27:54,784 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 06:27:54,784 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 06:27:54,785 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.44 seconds 2025-02-14 06:27:54,785 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:27:54,785 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21983.14 MB 2025-02-14 06:27:54,785 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22382.60 MB 2025-02-14 06:27:54,785 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 399.46 MB 2025-02-14 06:27:54,785 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28684.85 MB 2025-02-14 06:27:54,785 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25503.47 MB 2025-02-14 06:27:54,785 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3181.38 MB 2025-02-14 06:27:54,785 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26324.48 MB 2025-02-14 06:27:54,796 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 06:27:54,796 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 06:27:54,796 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:27:54,796 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:27:54,796 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22382.60 MB 2025-02-14 06:27:54,796 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23805.78 MB 2025-02-14 06:27:54,796 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1423.18 MB 2025-02-14 06:27:54,796 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25503.47 MB 2025-02-14 06:27:54,796 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27636.27 MB 2025-02-14 06:27:54,796 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2132.80 MB 2025-02-14 06:27:54,796 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24873.18 MB 2025-02-14 06:27:54,959 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 06:27:54,959 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 06:27:54,959 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 06:27:54,959 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:27:54,959 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23805.78 MB 2025-02-14 06:27:54,959 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25494.36 MB 2025-02-14 06:27:54,959 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1688.59 MB 2025-02-14 06:27:54,959 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27636.27 MB 2025-02-14 06:27:54,959 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31901.88 MB 2025-02-14 06:27:54,959 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4265.61 MB 2025-02-14 06:27:54,959 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29669.56 MB 2025-02-14 06:27:54,959 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 06:27:54,959 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 06:27:54,959 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 06:27:54,959 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:27:54,959 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22382.60 MB 2025-02-14 06:27:54,959 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25494.36 MB 2025-02-14 06:27:54,959 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3111.77 MB 2025-02-14 06:27:54,960 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25503.47 MB 2025-02-14 06:27:54,960 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31901.88 MB 2025-02-14 06:27:54,960 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6398.41 MB 2025-02-14 06:27:54,960 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29669.56 MB 2025-02-14 06:27:55,086 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 06:27:55,086 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 06:27:55,086 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 06:27:55,086 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:27:55,086 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26648.35 MB 2025-02-14 06:27:55,086 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27225.52 MB 2025-02-14 06:27:55,086 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 577.17 MB 2025-02-14 06:27:55,086 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31901.88 MB 2025-02-14 06:27:55,086 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32210.16 MB 2025-02-14 06:27:55,086 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 308.28 MB 2025-02-14 06:27:55,086 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27758.13 MB 2025-02-14 06:27:55,102 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 06:27:55,102 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 06:27:55,102 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:27:55,102 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:27:55,102 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27536.22 MB 2025-02-14 06:27:55,102 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27752.45 MB 2025-02-14 06:27:55,102 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 216.23 MB 2025-02-14 06:27:55,102 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32210.16 MB 2025-02-14 06:27:55,102 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32210.16 MB 2025-02-14 06:27:55,102 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:27:55,102 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27881.70 MB 2025-02-14 06:27:55,103 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 06:27:55,103 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 06:27:55,103 - resource_logging.py:150 - __exit__ - DEBUG - Time: 6.42 seconds 2025-02-14 06:27:55,103 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:27:55,103 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19352.98 MB 2025-02-14 06:27:55,103 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27953.52 MB 2025-02-14 06:27:55,103 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8600.54 MB 2025-02-14 06:27:55,103 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60087.60 MB 2025-02-14 06:27:55,103 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32210.16 MB 2025-02-14 06:27:55,103 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27877.44 MB 2025-02-14 06:27:55,103 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27953.52 MB 2025-02-14 06:27:55,370 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 06:27:55,370 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 06:27:55,370 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 06:27:55,370 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:27:55,370 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27953.52 MB 2025-02-14 06:27:55,370 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30967.56 MB 2025-02-14 06:27:55,370 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-14 06:27:55,370 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32210.16 MB 2025-02-14 06:27:55,370 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32210.16 MB 2025-02-14 06:27:55,370 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:27:55,370 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31268.92 MB 2025-02-14 06:27:55,388 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 06:27:55,388 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2 ('] 2025-02-14 06:27:55,394 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 06:27:55,395 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 06:27:55,395 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:27:55,395 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:27:55,395 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23889.73 MB 2025-02-14 06:27:55,395 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32328.76 MB 2025-02-14 06:27:55,395 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 06:27:55,395 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32210.16 MB 2025-02-14 06:27:55,395 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42700.11 MB 2025-02-14 06:27:55,395 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 06:27:55,395 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32328.76 MB 2025-02-14 06:27:55,557 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 06:27:55,559 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:27:55,559 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 06:27:55,560 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:27:55,560 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 06:27:55,564 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 06:27:55,565 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:27:55,565 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 06:27:55,566 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2 ('] 2025-02-14 06:28:45,891 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:28:45,891 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 06:28:45,896 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 06:28:45,900 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:28:45,900 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1933, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 06:28:45,901 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:28:45,901 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1933, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 06:29:15,741 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 06:29:15,742 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 06:29:15,742 - resource_logging.py:150 - __exit__ - DEBUG - Time: 29.83 seconds 2025-02-14 06:29:15,742 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:29:15,742 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31773.73 MB 2025-02-14 06:29:15,742 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38614.64 MB 2025-02-14 06:29:15,742 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6840.91 MB 2025-02-14 06:29:15,742 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55285.12 MB 2025-02-14 06:29:15,742 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45355.11 MB 2025-02-14 06:29:15,742 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9930.01 MB 2025-02-14 06:29:15,742 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47587.69 MB 2025-02-14 06:29:15,862 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 06:29:15,862 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 06:29:15,862 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 06:29:15,862 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:29:15,862 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38614.64 MB 2025-02-14 06:29:15,862 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31162.48 MB 2025-02-14 06:29:15,862 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7452.15 MB 2025-02-14 06:29:15,862 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45355.11 MB 2025-02-14 06:29:15,862 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 67540.88 MB 2025-02-14 06:29:15,862 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 22185.77 MB 2025-02-14 06:29:15,862 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 57980.97 MB 2025-02-14 06:29:17,800 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 06:29:17,800 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 06:29:17,800 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-14 06:29:17,800 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:29:17,800 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31162.48 MB 2025-02-14 06:29:17,800 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31693.32 MB 2025-02-14 06:29:17,800 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 06:29:17,800 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 67540.88 MB 2025-02-14 06:29:17,800 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35735.47 MB 2025-02-14 06:29:17,800 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31805.41 MB 2025-02-14 06:29:17,800 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35673.69 MB 2025-02-14 06:29:17,814 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 06:29:17,814 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 06:29:17,814 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:29:17,814 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:29:17,814 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31693.32 MB 2025-02-14 06:29:17,814 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33582.86 MB 2025-02-14 06:29:17,814 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 06:29:17,814 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35735.47 MB 2025-02-14 06:29:17,814 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38566.63 MB 2025-02-14 06:29:17,814 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-14 06:29:17,814 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35000.28 MB 2025-02-14 06:29:18,043 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 06:29:18,043 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 06:29:18,043 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 06:29:18,043 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:29:18,043 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33582.86 MB 2025-02-14 06:29:18,043 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35825.30 MB 2025-02-14 06:29:18,043 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2242.45 MB 2025-02-14 06:29:18,044 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38566.63 MB 2025-02-14 06:29:18,044 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44700.79 MB 2025-02-14 06:29:18,044 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 06:29:18,044 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41369.58 MB 2025-02-14 06:29:18,044 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 06:29:18,044 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 06:29:18,044 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.24 seconds 2025-02-14 06:29:18,044 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:29:18,044 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31693.32 MB 2025-02-14 06:29:18,044 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35825.30 MB 2025-02-14 06:29:18,044 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.98 MB 2025-02-14 06:29:18,044 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35735.47 MB 2025-02-14 06:29:18,044 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44700.79 MB 2025-02-14 06:29:18,044 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8965.32 MB 2025-02-14 06:29:18,044 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41369.58 MB 2025-02-14 06:29:18,213 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 06:29:18,214 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 06:29:18,214 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 06:29:18,214 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:29:18,214 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37358.84 MB 2025-02-14 06:29:18,214 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38125.85 MB 2025-02-14 06:29:18,214 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 06:29:18,214 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44700.79 MB 2025-02-14 06:29:18,214 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45113.93 MB 2025-02-14 06:29:18,214 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 06:29:18,214 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38833.63 MB 2025-02-14 06:29:18,233 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 06:29:18,233 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 06:29:18,233 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:29:18,233 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:29:18,233 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38538.73 MB 2025-02-14 06:29:18,233 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38766.22 MB 2025-02-14 06:29:18,233 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.49 MB 2025-02-14 06:29:18,233 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45113.93 MB 2025-02-14 06:29:18,233 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45113.93 MB 2025-02-14 06:29:18,233 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:29:18,233 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38974.66 MB 2025-02-14 06:29:18,234 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 06:29:18,234 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 06:29:18,234 - resource_logging.py:150 - __exit__ - DEBUG - Time: 32.33 seconds 2025-02-14 06:29:18,234 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:29:18,234 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25039.00 MB 2025-02-14 06:29:18,234 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38966.78 MB 2025-02-14 06:29:18,234 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13927.78 MB 2025-02-14 06:29:18,234 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55285.12 MB 2025-02-14 06:29:18,234 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45113.93 MB 2025-02-14 06:29:18,234 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10171.19 MB 2025-02-14 06:29:18,234 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38974.66 MB 2025-02-14 06:29:18,503 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 06:29:18,503 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 06:29:18,503 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 06:29:18,503 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:29:18,503 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38966.78 MB 2025-02-14 06:29:18,503 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30035.46 MB 2025-02-14 06:29:18,503 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8931.32 MB 2025-02-14 06:29:18,503 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45113.93 MB 2025-02-14 06:29:18,503 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45113.93 MB 2025-02-14 06:29:18,503 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:29:18,503 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41471.99 MB 2025-02-14 06:29:18,521 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8141, cut from 8143 2025-02-14 06:29:18,521 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 06:29:18,527 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 06:29:18,527 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 06:29:18,527 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:29:18,527 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:29:18,527 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30035.46 MB 2025-02-14 06:29:18,527 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38452.58 MB 2025-02-14 06:29:18,527 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8417.12 MB 2025-02-14 06:29:18,527 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45113.93 MB 2025-02-14 06:29:18,527 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49297.75 MB 2025-02-14 06:29:18,527 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4183.82 MB 2025-02-14 06:29:18,528 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38452.58 MB 2025-02-14 06:29:18,690 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7933] 2025-02-14 06:29:18,691 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:29:18,691 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 06:29:18,692 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:29:18,692 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 06:29:18,697 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 06:29:18,698 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:29:18,698 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 06:29:18,698 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 06:29:32,321 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:29:32,321 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 06:29:32,326 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 06:29:32,330 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:29:32,330 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1190, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 06:29:32,331 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:29:32,331 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1190, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 06:29:50,956 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 06:29:50,957 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 06:29:50,957 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.62 seconds 2025-02-14 06:29:50,957 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:29:50,957 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26596.38 MB 2025-02-14 06:29:50,957 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30807.73 MB 2025-02-14 06:29:50,957 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4211.34 MB 2025-02-14 06:29:50,957 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57665.39 MB 2025-02-14 06:29:50,957 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34315.70 MB 2025-02-14 06:29:50,957 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23349.69 MB 2025-02-14 06:29:50,957 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39692.44 MB 2025-02-14 06:29:51,039 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 06:29:51,040 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 06:29:51,040 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 06:29:51,040 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:29:51,040 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30807.73 MB 2025-02-14 06:29:51,040 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27300.90 MB 2025-02-14 06:29:51,040 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3506.82 MB 2025-02-14 06:29:51,040 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34315.70 MB 2025-02-14 06:29:51,040 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51422.17 MB 2025-02-14 06:29:51,040 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 17106.47 MB 2025-02-14 06:29:51,040 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43385.47 MB 2025-02-14 06:29:52,974 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 06:29:52,974 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 06:29:52,974 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 06:29:52,974 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:29:52,974 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27300.90 MB 2025-02-14 06:29:52,974 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27831.75 MB 2025-02-14 06:29:52,974 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 06:29:52,974 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51422.17 MB 2025-02-14 06:29:52,974 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32226.93 MB 2025-02-14 06:29:52,974 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19195.23 MB 2025-02-14 06:29:52,974 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31812.12 MB 2025-02-14 06:29:52,987 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 06:29:52,987 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 06:29:52,987 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:29:52,987 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:29:52,987 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27831.75 MB 2025-02-14 06:29:52,987 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29721.28 MB 2025-02-14 06:29:52,987 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 06:29:52,988 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32226.93 MB 2025-02-14 06:29:52,988 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34114.37 MB 2025-02-14 06:29:52,988 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 06:29:52,988 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31138.71 MB 2025-02-14 06:29:53,198 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 06:29:53,198 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 06:29:53,198 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 06:29:53,198 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:29:53,198 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29721.28 MB 2025-02-14 06:29:53,198 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31963.73 MB 2025-02-14 06:29:53,198 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2242.45 MB 2025-02-14 06:29:53,198 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34114.37 MB 2025-02-14 06:29:53,198 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40248.54 MB 2025-02-14 06:29:53,198 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 06:29:53,198 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37508.01 MB 2025-02-14 06:29:53,199 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 06:29:53,199 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 06:29:53,199 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 06:29:53,199 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:29:53,199 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27831.75 MB 2025-02-14 06:29:53,199 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31963.73 MB 2025-02-14 06:29:53,199 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.98 MB 2025-02-14 06:29:53,199 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32226.93 MB 2025-02-14 06:29:53,199 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40248.54 MB 2025-02-14 06:29:53,199 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-14 06:29:53,199 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37508.01 MB 2025-02-14 06:29:53,366 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 06:29:53,366 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 06:29:53,366 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 06:29:53,366 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:29:53,366 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33497.27 MB 2025-02-14 06:29:53,366 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34264.27 MB 2025-02-14 06:29:53,366 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 06:29:53,366 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40248.54 MB 2025-02-14 06:29:53,366 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40661.68 MB 2025-02-14 06:29:53,366 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 06:29:53,366 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34972.06 MB 2025-02-14 06:29:53,385 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 06:29:53,385 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 06:29:53,385 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:29:53,385 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:29:53,385 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34677.16 MB 2025-02-14 06:29:53,385 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34905.62 MB 2025-02-14 06:29:53,385 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.46 MB 2025-02-14 06:29:53,385 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40661.68 MB 2025-02-14 06:29:53,385 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40661.68 MB 2025-02-14 06:29:53,385 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:29:53,385 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35140.18 MB 2025-02-14 06:29:53,386 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 06:29:53,386 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 06:29:53,386 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.05 seconds 2025-02-14 06:29:53,386 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:29:53,386 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22450.33 MB 2025-02-14 06:29:53,386 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35106.15 MB 2025-02-14 06:29:53,386 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12655.82 MB 2025-02-14 06:29:53,386 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57665.39 MB 2025-02-14 06:29:53,386 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40661.68 MB 2025-02-14 06:29:53,386 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17003.71 MB 2025-02-14 06:29:53,386 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35140.18 MB 2025-02-14 06:29:53,654 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 06:29:53,654 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 06:29:53,655 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 06:29:53,655 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:29:53,655 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35106.15 MB 2025-02-14 06:29:53,655 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27446.42 MB 2025-02-14 06:29:53,655 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7659.73 MB 2025-02-14 06:29:53,655 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40661.68 MB 2025-02-14 06:29:53,655 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40661.68 MB 2025-02-14 06:29:53,655 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:29:53,655 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37611.06 MB 2025-02-14 06:29:53,672 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8140, cut from 8142 2025-02-14 06:29:53,673 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 06:29:53,679 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 06:29:53,679 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 06:29:53,679 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:29:53,679 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:29:53,679 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27446.42 MB 2025-02-14 06:29:53,679 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35863.03 MB 2025-02-14 06:29:53,679 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8416.60 MB 2025-02-14 06:29:53,679 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40661.68 MB 2025-02-14 06:29:53,679 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49029.32 MB 2025-02-14 06:29:53,679 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8367.64 MB 2025-02-14 06:29:53,679 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35863.03 MB 2025-02-14 06:29:53,841 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7932] 2025-02-14 06:29:53,842 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:29:53,842 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 06:29:53,843 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:29:53,843 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 06:29:53,848 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 06:29:53,849 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:29:53,849 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 06:29:53,849 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 06:30:57,264 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:30:57,265 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 06:30:57,269 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 06:30:57,273 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:30:57,273 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 198, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 06:30:57,274 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:30:57,274 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 198, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 06:31:00,369 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 06:31:00,369 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 06:31:00,369 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.09 seconds 2025-02-14 06:31:00,369 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:31:00,369 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19683.97 MB 2025-02-14 06:31:00,369 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20384.68 MB 2025-02-14 06:31:00,369 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 700.71 MB 2025-02-14 06:31:00,369 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57396.95 MB 2025-02-14 06:31:00,369 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22909.29 MB 2025-02-14 06:31:00,369 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34487.66 MB 2025-02-14 06:31:00,369 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29381.83 MB 2025-02-14 06:31:00,383 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 06:31:00,384 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 06:31:00,384 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:31:00,384 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:31:00,384 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20384.68 MB 2025-02-14 06:31:00,384 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20724.42 MB 2025-02-14 06:31:00,384 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 339.74 MB 2025-02-14 06:31:00,384 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22909.29 MB 2025-02-14 06:31:00,384 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24654.12 MB 2025-02-14 06:31:00,384 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1744.83 MB 2025-02-14 06:31:00,384 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23187.34 MB 2025-02-14 06:31:01,340 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 06:31:01,340 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 06:31:01,340 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.95 seconds 2025-02-14 06:31:01,340 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:31:01,340 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20724.42 MB 2025-02-14 06:31:01,340 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20987.18 MB 2025-02-14 06:31:01,340 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 262.77 MB 2025-02-14 06:31:01,340 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24654.12 MB 2025-02-14 06:31:01,340 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23249.03 MB 2025-02-14 06:31:01,340 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1405.09 MB 2025-02-14 06:31:01,340 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24980.83 MB 2025-02-14 06:31:01,349 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 06:31:01,349 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 06:31:01,349 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:31:01,349 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:31:01,349 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20987.12 MB 2025-02-14 06:31:01,349 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21923.26 MB 2025-02-14 06:31:01,349 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 936.14 MB 2025-02-14 06:31:01,349 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23249.03 MB 2025-02-14 06:31:01,349 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24184.36 MB 2025-02-14 06:31:01,349 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 935.33 MB 2025-02-14 06:31:01,349 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22624.89 MB 2025-02-14 06:31:01,456 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 06:31:01,456 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 06:31:01,456 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 06:31:01,456 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:31:01,456 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21923.26 MB 2025-02-14 06:31:01,456 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23033.54 MB 2025-02-14 06:31:01,456 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1110.28 MB 2025-02-14 06:31:01,456 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24184.36 MB 2025-02-14 06:31:01,456 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27225.23 MB 2025-02-14 06:31:01,456 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3040.87 MB 2025-02-14 06:31:01,456 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25780.02 MB 2025-02-14 06:31:01,456 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 06:31:01,456 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 06:31:01,456 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 06:31:01,456 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:31:01,457 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20987.12 MB 2025-02-14 06:31:01,457 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23033.54 MB 2025-02-14 06:31:01,457 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2046.42 MB 2025-02-14 06:31:01,457 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23249.03 MB 2025-02-14 06:31:01,457 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27225.23 MB 2025-02-14 06:31:01,457 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3976.20 MB 2025-02-14 06:31:01,457 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25780.02 MB 2025-02-14 06:31:01,542 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 06:31:01,542 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 06:31:01,542 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 06:31:01,542 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:31:01,542 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23792.64 MB 2025-02-14 06:31:01,542 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24172.31 MB 2025-02-14 06:31:01,542 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 379.67 MB 2025-02-14 06:31:01,542 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27225.23 MB 2025-02-14 06:31:01,542 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27426.55 MB 2025-02-14 06:31:01,542 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 201.33 MB 2025-02-14 06:31:01,542 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24525.98 MB 2025-02-14 06:31:01,553 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 06:31:01,553 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 06:31:01,553 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:31:01,553 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:31:01,553 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24376.69 MB 2025-02-14 06:31:01,553 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24596.52 MB 2025-02-14 06:31:01,553 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 219.83 MB 2025-02-14 06:31:01,553 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27426.55 MB 2025-02-14 06:31:01,553 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27426.55 MB 2025-02-14 06:31:01,553 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:31:01,553 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24651.95 MB 2025-02-14 06:31:01,554 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 06:31:01,554 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 06:31:01,554 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.28 seconds 2025-02-14 06:31:01,554 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:31:01,554 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18994.12 MB 2025-02-14 06:31:01,554 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24797.30 MB 2025-02-14 06:31:01,554 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5803.18 MB 2025-02-14 06:31:01,554 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57396.95 MB 2025-02-14 06:31:01,554 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27426.55 MB 2025-02-14 06:31:01,554 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29970.40 MB 2025-02-14 06:31:01,554 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24797.30 MB 2025-02-14 06:31:01,820 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 06:31:01,820 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 06:31:01,820 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 06:31:01,820 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:31:01,820 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20030.70 MB 2025-02-14 06:31:01,820 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23040.31 MB 2025-02-14 06:31:01,820 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3009.61 MB 2025-02-14 06:31:01,820 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27426.55 MB 2025-02-14 06:31:01,820 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27426.55 MB 2025-02-14 06:31:01,820 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:31:01,820 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23341.24 MB 2025-02-14 06:31:01,838 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8150, cut from 8152 2025-02-14 06:31:01,838 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 video rate for this video is 2,'] 2025-02-14 06:31:01,844 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 06:31:01,844 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 06:31:01,844 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:31:01,844 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:31:01,844 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23040.31 MB 2025-02-14 06:31:01,844 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31466.81 MB 2025-02-14 06:31:01,844 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8426.50 MB 2025-02-14 06:31:01,844 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27426.55 MB 2025-02-14 06:31:01,844 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37899.73 MB 2025-02-14 06:31:01,844 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10473.18 MB 2025-02-14 06:31:01,844 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31466.81 MB 2025-02-14 06:31:01,995 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7942] 2025-02-14 06:31:01,996 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:31:01,996 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 06:31:01,997 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:31:01,997 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 06:31:02,002 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 06:31:02,003 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:31:02,003 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 06:31:02,003 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 video rate for this video is 2,'] 2025-02-14 06:31:40,552 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:31:40,553 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 06:31:40,558 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 06:31:40,561 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:31:40,561 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1708, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 06:31:40,562 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:31:40,562 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1708, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 06:32:06,916 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 06:32:06,917 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 06:32:06,917 - resource_logging.py:150 - __exit__ - DEBUG - Time: 26.35 seconds 2025-02-14 06:32:06,917 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:32:06,917 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30205.89 MB 2025-02-14 06:32:06,917 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36250.41 MB 2025-02-14 06:32:06,917 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6044.52 MB 2025-02-14 06:32:06,917 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50465.87 MB 2025-02-14 06:32:06,917 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44533.02 MB 2025-02-14 06:32:06,917 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5932.84 MB 2025-02-14 06:32:06,917 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45113.89 MB 2025-02-14 06:32:07,015 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 06:32:07,015 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 06:32:07,015 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 06:32:07,015 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:32:07,015 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36250.41 MB 2025-02-14 06:32:07,015 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29992.78 MB 2025-02-14 06:32:07,016 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6257.63 MB 2025-02-14 06:32:07,016 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44533.02 MB 2025-02-14 06:32:07,016 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54439.97 MB 2025-02-14 06:32:07,016 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9906.95 MB 2025-02-14 06:32:07,016 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47590.57 MB 2025-02-14 06:32:08,959 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 06:32:08,959 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 06:32:08,959 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-14 06:32:08,959 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:32:08,959 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29992.78 MB 2025-02-14 06:32:08,959 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30523.62 MB 2025-02-14 06:32:08,959 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 06:32:08,959 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54439.97 MB 2025-02-14 06:32:08,959 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35714.50 MB 2025-02-14 06:32:08,959 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18725.47 MB 2025-02-14 06:32:08,959 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34502.95 MB 2025-02-14 06:32:08,974 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 06:32:08,974 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 06:32:08,974 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:32:08,974 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:32:08,974 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30523.62 MB 2025-02-14 06:32:08,974 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32413.15 MB 2025-02-14 06:32:08,974 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 06:32:08,974 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35714.50 MB 2025-02-14 06:32:08,974 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37601.94 MB 2025-02-14 06:32:08,974 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 06:32:08,974 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33830.58 MB 2025-02-14 06:32:09,184 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 06:32:09,184 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 06:32:09,184 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 06:32:09,184 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:32:09,184 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32413.15 MB 2025-02-14 06:32:09,184 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34655.60 MB 2025-02-14 06:32:09,184 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2242.45 MB 2025-02-14 06:32:09,184 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37601.94 MB 2025-02-14 06:32:09,184 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43736.10 MB 2025-02-14 06:32:09,184 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 06:32:09,184 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40199.88 MB 2025-02-14 06:32:09,184 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 06:32:09,184 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 06:32:09,184 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 06:32:09,184 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:32:09,184 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30523.62 MB 2025-02-14 06:32:09,185 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34655.60 MB 2025-02-14 06:32:09,185 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.98 MB 2025-02-14 06:32:09,185 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35714.50 MB 2025-02-14 06:32:09,185 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43736.10 MB 2025-02-14 06:32:09,185 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-14 06:32:09,185 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40199.88 MB 2025-02-14 06:32:09,352 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 06:32:09,352 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 06:32:09,352 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 06:32:09,352 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:32:09,352 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36189.14 MB 2025-02-14 06:32:09,352 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36956.14 MB 2025-02-14 06:32:09,352 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 06:32:09,352 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43736.10 MB 2025-02-14 06:32:09,352 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44151.34 MB 2025-02-14 06:32:09,352 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 06:32:09,352 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37663.93 MB 2025-02-14 06:32:09,371 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 06:32:09,371 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 06:32:09,371 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:32:09,371 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:32:09,371 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37369.03 MB 2025-02-14 06:32:09,371 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37596.17 MB 2025-02-14 06:32:09,371 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.14 MB 2025-02-14 06:32:09,371 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44151.34 MB 2025-02-14 06:32:09,371 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44151.34 MB 2025-02-14 06:32:09,371 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:32:09,371 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37828.74 MB 2025-02-14 06:32:09,372 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 06:32:09,372 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 06:32:09,372 - resource_logging.py:150 - __exit__ - DEBUG - Time: 28.81 seconds 2025-02-14 06:32:09,372 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:32:09,372 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24255.08 MB 2025-02-14 06:32:09,372 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37796.98 MB 2025-02-14 06:32:09,372 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13541.89 MB 2025-02-14 06:32:09,372 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50465.87 MB 2025-02-14 06:32:09,372 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44151.34 MB 2025-02-14 06:32:09,372 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6314.52 MB 2025-02-14 06:32:09,372 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37828.74 MB 2025-02-14 06:32:09,642 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 06:32:09,643 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 06:32:09,643 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 06:32:09,643 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:32:09,643 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37796.98 MB 2025-02-14 06:32:09,643 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29255.28 MB 2025-02-14 06:32:09,643 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8541.69 MB 2025-02-14 06:32:09,643 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44151.34 MB 2025-02-14 06:32:09,643 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44151.34 MB 2025-02-14 06:32:09,643 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:32:09,643 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40305.26 MB 2025-02-14 06:32:09,661 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8151, cut from 8153 2025-02-14 06:32:09,661 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 06:32:09,667 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 06:32:09,667 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 06:32:09,667 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:32:09,667 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:32:09,667 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29255.28 MB 2025-02-14 06:32:09,667 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37682.62 MB 2025-02-14 06:32:09,667 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8427.34 MB 2025-02-14 06:32:09,667 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44151.34 MB 2025-02-14 06:32:09,667 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52531.56 MB 2025-02-14 06:32:09,667 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8380.22 MB 2025-02-14 06:32:09,667 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37682.62 MB 2025-02-14 06:32:09,833 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7943] 2025-02-14 06:32:09,834 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:32:09,834 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 06:32:09,835 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:32:09,835 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 06:32:09,840 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 06:32:09,841 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:32:09,841 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 06:32:09,841 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 06:32:25,502 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:32:25,502 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 06:32:25,507 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 06:32:25,511 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:32:25,511 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1071, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 06:32:25,512 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:32:25,512 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1071, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 06:32:42,236 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 06:32:42,237 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 06:32:42,237 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.72 seconds 2025-02-14 06:32:42,237 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:32:42,237 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25767.17 MB 2025-02-14 06:32:42,237 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29557.38 MB 2025-02-14 06:32:42,237 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3790.21 MB 2025-02-14 06:32:42,237 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60911.78 MB 2025-02-14 06:32:42,237 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33904.66 MB 2025-02-14 06:32:42,237 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27007.12 MB 2025-02-14 06:32:42,237 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38410.24 MB 2025-02-14 06:32:42,305 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 06:32:42,305 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 06:32:42,305 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 06:32:42,305 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:32:42,305 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29557.38 MB 2025-02-14 06:32:42,305 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26682.26 MB 2025-02-14 06:32:42,305 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2875.12 MB 2025-02-14 06:32:42,305 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33904.66 MB 2025-02-14 06:32:42,305 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45929.73 MB 2025-02-14 06:32:42,305 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 12025.07 MB 2025-02-14 06:32:42,305 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40122.45 MB 2025-02-14 06:32:44,233 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 06:32:44,233 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 06:32:44,233 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 06:32:44,233 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:32:44,233 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26682.26 MB 2025-02-14 06:32:44,233 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27213.10 MB 2025-02-14 06:32:44,233 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 06:32:44,233 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45929.73 MB 2025-02-14 06:32:44,233 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32237.42 MB 2025-02-14 06:32:44,233 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13692.31 MB 2025-02-14 06:32:44,233 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31192.44 MB 2025-02-14 06:32:44,247 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 06:32:44,247 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 06:32:44,247 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:32:44,247 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:32:44,247 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27213.10 MB 2025-02-14 06:32:44,247 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29102.64 MB 2025-02-14 06:32:44,247 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 06:32:44,247 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32237.42 MB 2025-02-14 06:32:44,247 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34124.86 MB 2025-02-14 06:32:44,247 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 06:32:44,247 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30520.06 MB 2025-02-14 06:32:44,455 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 06:32:44,455 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 06:32:44,455 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 06:32:44,455 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:32:44,455 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29102.64 MB 2025-02-14 06:32:44,455 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31345.08 MB 2025-02-14 06:32:44,455 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2242.45 MB 2025-02-14 06:32:44,455 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34124.86 MB 2025-02-14 06:32:44,455 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40259.03 MB 2025-02-14 06:32:44,455 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 06:32:44,455 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36889.36 MB 2025-02-14 06:32:44,456 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 06:32:44,456 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 06:32:44,456 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 06:32:44,456 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:32:44,456 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27213.10 MB 2025-02-14 06:32:44,456 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31345.08 MB 2025-02-14 06:32:44,456 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.98 MB 2025-02-14 06:32:44,456 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32237.42 MB 2025-02-14 06:32:44,456 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40259.03 MB 2025-02-14 06:32:44,456 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-14 06:32:44,456 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36889.36 MB 2025-02-14 06:32:44,627 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 06:32:44,627 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 06:32:44,627 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 06:32:44,627 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:32:44,627 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32878.62 MB 2025-02-14 06:32:44,627 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33645.63 MB 2025-02-14 06:32:44,627 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 06:32:44,627 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40259.03 MB 2025-02-14 06:32:44,627 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40672.17 MB 2025-02-14 06:32:44,627 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 06:32:44,627 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34353.41 MB 2025-02-14 06:32:44,648 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 06:32:44,648 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 06:32:44,648 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:32:44,648 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:32:44,648 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34058.51 MB 2025-02-14 06:32:44,648 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34287.02 MB 2025-02-14 06:32:44,648 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.51 MB 2025-02-14 06:32:44,648 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40672.17 MB 2025-02-14 06:32:44,648 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40672.17 MB 2025-02-14 06:32:44,648 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:32:44,648 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34488.38 MB 2025-02-14 06:32:44,649 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 06:32:44,649 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 06:32:44,649 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.13 seconds 2025-02-14 06:32:44,649 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:32:44,649 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22035.72 MB 2025-02-14 06:32:44,649 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34488.09 MB 2025-02-14 06:32:44,649 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12452.37 MB 2025-02-14 06:32:44,650 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60911.78 MB 2025-02-14 06:32:44,650 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40672.17 MB 2025-02-14 06:32:44,650 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20239.61 MB 2025-02-14 06:32:44,650 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34488.38 MB 2025-02-14 06:32:44,920 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 06:32:44,920 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 06:32:44,920 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 06:32:44,920 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:32:44,920 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34488.09 MB 2025-02-14 06:32:44,920 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27040.11 MB 2025-02-14 06:32:44,920 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7447.98 MB 2025-02-14 06:32:44,920 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40672.17 MB 2025-02-14 06:32:44,920 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40672.17 MB 2025-02-14 06:32:44,920 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:32:44,920 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36999.76 MB 2025-02-14 06:32:44,938 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 06:32:44,938 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 06:32:44,945 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 06:32:44,945 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 06:32:44,945 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:32:44,945 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:32:44,945 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27040.11 MB 2025-02-14 06:32:44,945 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35479.13 MB 2025-02-14 06:32:44,945 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 06:32:44,945 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40672.17 MB 2025-02-14 06:32:44,945 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49062.87 MB 2025-02-14 06:32:44,945 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 06:32:44,945 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35479.13 MB 2025-02-14 06:32:45,096 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 06:32:45,097 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:32:45,097 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 06:32:45,098 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:32:45,098 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 06:32:45,102 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 06:32:45,103 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:32:45,103 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 06:32:45,104 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 06:33:15,327 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:33:15,327 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 06:33:15,332 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 06:33:15,336 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:33:15,336 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 282, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 06:33:15,338 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:33:15,338 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 282, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 06:33:19,749 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 06:33:19,749 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 06:33:19,749 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.41 seconds 2025-02-14 06:33:19,749 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:33:19,749 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20269.29 MB 2025-02-14 06:33:19,749 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21267.54 MB 2025-02-14 06:33:19,749 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 998.24 MB 2025-02-14 06:33:19,749 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61647.88 MB 2025-02-14 06:33:19,749 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24383.59 MB 2025-02-14 06:33:19,749 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -37264.29 MB 2025-02-14 06:33:19,749 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30194.46 MB 2025-02-14 06:33:19,765 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 06:33:19,765 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 06:33:19,765 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:33:19,765 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:33:19,765 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21267.54 MB 2025-02-14 06:33:19,765 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20753.53 MB 2025-02-14 06:33:19,765 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -514.01 MB 2025-02-14 06:33:19,765 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24383.59 MB 2025-02-14 06:33:19,765 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25377.64 MB 2025-02-14 06:33:19,765 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 994.05 MB 2025-02-14 06:33:19,765 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23269.44 MB 2025-02-14 06:33:20,447 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 06:33:20,448 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 06:33:20,448 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.68 seconds 2025-02-14 06:33:20,448 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:33:20,448 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20753.53 MB 2025-02-14 06:33:20,448 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20939.32 MB 2025-02-14 06:33:20,448 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 185.79 MB 2025-02-14 06:33:20,448 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25377.64 MB 2025-02-14 06:33:20,448 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23882.37 MB 2025-02-14 06:33:20,448 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1495.27 MB 2025-02-14 06:33:20,448 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24925.00 MB 2025-02-14 06:33:20,455 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 06:33:20,455 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 06:33:20,455 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 06:33:20,455 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:33:20,455 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20939.25 MB 2025-02-14 06:33:20,455 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21600.43 MB 2025-02-14 06:33:20,455 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 661.18 MB 2025-02-14 06:33:20,455 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23882.37 MB 2025-02-14 06:33:20,455 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23882.37 MB 2025-02-14 06:33:20,455 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:33:20,455 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22096.54 MB 2025-02-14 06:33:20,533 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 06:33:20,533 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 06:33:20,533 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 06:33:20,533 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:33:20,533 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21600.43 MB 2025-02-14 06:33:20,533 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22385.12 MB 2025-02-14 06:33:20,533 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 784.69 MB 2025-02-14 06:33:20,533 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23882.37 MB 2025-02-14 06:33:20,533 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25704.79 MB 2025-02-14 06:33:20,533 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1822.43 MB 2025-02-14 06:33:20,533 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24330.30 MB 2025-02-14 06:33:20,534 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 06:33:20,534 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 06:33:20,534 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 06:33:20,534 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:33:20,534 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20939.25 MB 2025-02-14 06:33:20,534 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22385.12 MB 2025-02-14 06:33:20,534 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1445.87 MB 2025-02-14 06:33:20,534 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23882.37 MB 2025-02-14 06:33:20,534 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25704.79 MB 2025-02-14 06:33:20,534 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1822.43 MB 2025-02-14 06:33:20,534 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24330.30 MB 2025-02-14 06:33:20,597 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 06:33:20,597 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 06:33:20,597 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 06:33:20,597 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:33:20,597 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22921.86 MB 2025-02-14 06:33:20,597 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23190.84 MB 2025-02-14 06:33:20,597 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 268.98 MB 2025-02-14 06:33:20,597 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25704.79 MB 2025-02-14 06:33:20,597 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25847.40 MB 2025-02-14 06:33:20,598 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 142.61 MB 2025-02-14 06:33:20,598 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23448.28 MB 2025-02-14 06:33:20,607 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 06:33:20,607 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 06:33:20,607 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:33:20,607 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:33:20,607 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23335.36 MB 2025-02-14 06:33:20,607 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23547.72 MB 2025-02-14 06:33:20,607 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 212.36 MB 2025-02-14 06:33:20,607 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25847.40 MB 2025-02-14 06:33:20,607 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25847.40 MB 2025-02-14 06:33:20,607 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:33:20,607 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23553.26 MB 2025-02-14 06:33:20,608 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 06:33:20,608 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 06:33:20,608 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.27 seconds 2025-02-14 06:33:20,608 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:33:20,608 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19286.78 MB 2025-02-14 06:33:20,608 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23748.50 MB 2025-02-14 06:33:20,608 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4461.71 MB 2025-02-14 06:33:20,608 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61647.88 MB 2025-02-14 06:33:20,608 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25847.40 MB 2025-02-14 06:33:20,608 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35800.48 MB 2025-02-14 06:33:20,608 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23748.50 MB 2025-02-14 06:33:20,879 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 06:33:20,879 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 06:33:20,879 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 06:33:20,879 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:33:20,879 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23748.50 MB 2025-02-14 06:33:20,879 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26758.11 MB 2025-02-14 06:33:20,879 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3009.61 MB 2025-02-14 06:33:20,879 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25847.40 MB 2025-02-14 06:33:20,879 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28800.19 MB 2025-02-14 06:33:20,879 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2952.79 MB 2025-02-14 06:33:20,879 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27059.49 MB 2025-02-14 06:33:20,897 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8150, cut from 8152 2025-02-14 06:33:20,897 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 06:33:20,903 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 06:33:20,903 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 06:33:20,903 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:33:20,903 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:33:20,903 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26758.11 MB 2025-02-14 06:33:20,904 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35184.61 MB 2025-02-14 06:33:20,904 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8426.50 MB 2025-02-14 06:33:20,904 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28800.19 MB 2025-02-14 06:33:20,904 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39273.37 MB 2025-02-14 06:33:20,904 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10473.18 MB 2025-02-14 06:33:20,904 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35184.61 MB 2025-02-14 06:33:21,066 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7942] 2025-02-14 06:33:21,067 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:33:21,067 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 06:33:21,068 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:33:21,068 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 06:33:21,073 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 06:33:21,074 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:33:21,074 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 06:33:21,074 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 06:33:46,899 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:33:46,899 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 06:33:46,904 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 06:33:46,907 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:33:46,907 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 582, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 06:33:46,908 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:33:46,908 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 582, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 06:33:55,900 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 06:33:55,900 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 06:33:55,900 - resource_logging.py:150 - __exit__ - DEBUG - Time: 8.99 seconds 2025-02-14 06:33:55,900 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:33:55,900 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31927.56 MB 2025-02-14 06:33:55,900 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33987.23 MB 2025-02-14 06:33:55,900 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2059.67 MB 2025-02-14 06:33:55,900 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51841.60 MB 2025-02-14 06:33:55,900 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37388.03 MB 2025-02-14 06:33:55,900 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14453.57 MB 2025-02-14 06:33:55,900 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42985.18 MB 2025-02-14 06:33:55,939 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 06:33:55,939 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 06:33:55,939 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-14 06:33:55,939 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:33:55,939 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33987.23 MB 2025-02-14 06:33:55,939 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33706.87 MB 2025-02-14 06:33:55,940 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -280.35 MB 2025-02-14 06:33:55,940 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37388.03 MB 2025-02-14 06:33:55,940 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44648.37 MB 2025-02-14 06:33:55,940 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7260.34 MB 2025-02-14 06:33:55,940 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42026.46 MB 2025-02-14 06:33:57,873 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 06:33:57,873 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 06:33:57,873 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 06:33:57,873 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:33:57,873 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33706.87 MB 2025-02-14 06:33:57,873 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34237.71 MB 2025-02-14 06:33:57,873 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 06:33:57,873 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44648.37 MB 2025-02-14 06:33:57,873 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36742.10 MB 2025-02-14 06:33:57,873 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7906.26 MB 2025-02-14 06:33:57,873 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38218.08 MB 2025-02-14 06:33:57,887 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 06:33:57,888 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 06:33:57,888 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:33:57,888 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:33:57,888 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34237.71 MB 2025-02-14 06:33:57,888 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36126.93 MB 2025-02-14 06:33:57,888 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.22 MB 2025-02-14 06:33:57,888 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36742.10 MB 2025-02-14 06:33:57,888 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39575.36 MB 2025-02-14 06:33:57,888 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2833.25 MB 2025-02-14 06:33:57,888 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37544.36 MB 2025-02-14 06:33:58,094 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 06:33:58,094 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 06:33:58,094 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 06:33:58,094 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:33:58,094 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36126.93 MB 2025-02-14 06:33:58,094 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38368.79 MB 2025-02-14 06:33:58,094 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 06:33:58,094 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39575.36 MB 2025-02-14 06:33:58,094 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46181.38 MB 2025-02-14 06:33:58,094 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 06:33:58,094 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43913.07 MB 2025-02-14 06:33:58,095 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 06:33:58,095 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 06:33:58,095 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 06:33:58,095 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:33:58,095 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34237.71 MB 2025-02-14 06:33:58,095 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38368.79 MB 2025-02-14 06:33:58,095 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.07 MB 2025-02-14 06:33:58,095 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36742.10 MB 2025-02-14 06:33:58,095 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46181.38 MB 2025-02-14 06:33:58,095 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9439.28 MB 2025-02-14 06:33:58,095 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43913.07 MB 2025-02-14 06:33:58,255 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 06:33:58,255 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 06:33:58,255 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 06:33:58,255 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:33:58,255 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39902.33 MB 2025-02-14 06:33:58,255 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40669.33 MB 2025-02-14 06:33:58,255 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 06:33:58,255 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46181.38 MB 2025-02-14 06:33:58,255 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46598.72 MB 2025-02-14 06:33:58,255 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 06:33:58,255 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41377.12 MB 2025-02-14 06:33:58,274 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 06:33:58,274 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 06:33:58,274 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:33:58,274 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:33:58,274 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41082.22 MB 2025-02-14 06:33:58,274 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41309.60 MB 2025-02-14 06:33:58,274 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.38 MB 2025-02-14 06:33:58,274 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46598.72 MB 2025-02-14 06:33:58,274 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46600.81 MB 2025-02-14 06:33:58,274 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-14 06:33:58,274 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41484.96 MB 2025-02-14 06:33:58,275 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 06:33:58,275 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 06:33:58,275 - resource_logging.py:150 - __exit__ - DEBUG - Time: 11.36 seconds 2025-02-14 06:33:58,275 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:33:58,275 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29899.82 MB 2025-02-14 06:33:58,275 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41510.64 MB 2025-02-14 06:33:58,275 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11610.82 MB 2025-02-14 06:33:58,275 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51841.60 MB 2025-02-14 06:33:58,275 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46600.81 MB 2025-02-14 06:33:58,275 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5240.78 MB 2025-02-14 06:33:58,275 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41510.64 MB 2025-02-14 06:33:58,546 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 06:33:58,546 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 06:33:58,546 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 06:33:58,546 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:33:58,546 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41510.64 MB 2025-02-14 06:33:58,546 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34903.83 MB 2025-02-14 06:33:58,546 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6606.81 MB 2025-02-14 06:33:58,546 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46600.81 MB 2025-02-14 06:33:58,546 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46600.81 MB 2025-02-14 06:33:58,546 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:33:58,546 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44022.00 MB 2025-02-14 06:33:58,564 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8161, cut from 8163 2025-02-14 06:33:58,564 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 06:33:58,571 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 06:33:58,571 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 06:33:58,571 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:33:58,571 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:33:58,571 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34903.83 MB 2025-02-14 06:33:58,571 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43342.67 MB 2025-02-14 06:33:58,571 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8438.84 MB 2025-02-14 06:33:58,571 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46600.81 MB 2025-02-14 06:33:58,571 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54989.42 MB 2025-02-14 06:33:58,571 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8388.61 MB 2025-02-14 06:33:58,571 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43342.67 MB 2025-02-14 06:33:58,727 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7953] 2025-02-14 06:33:58,728 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:33:58,728 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 06:33:58,729 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:33:58,729 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 06:33:58,734 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 06:33:58,735 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:33:58,735 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 06:33:58,735 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 06:34:06,978 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:34:06,978 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 06:34:06,983 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 06:34:06,986 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:34:06,986 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 637, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 06:34:06,987 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:34:06,987 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 637, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 06:34:16,892 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 06:34:16,893 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 06:34:16,893 - resource_logging.py:150 - __exit__ - DEBUG - Time: 9.90 seconds 2025-02-14 06:34:16,893 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:34:16,893 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32310.81 MB 2025-02-14 06:34:16,893 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34565.25 MB 2025-02-14 06:34:16,893 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2254.44 MB 2025-02-14 06:34:16,893 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63378.03 MB 2025-02-14 06:34:16,893 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39000.74 MB 2025-02-14 06:34:16,893 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24377.29 MB 2025-02-14 06:34:16,893 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43367.63 MB 2025-02-14 06:34:16,935 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 06:34:16,935 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 06:34:16,935 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-14 06:34:16,935 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:34:16,935 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34565.25 MB 2025-02-14 06:34:16,935 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33992.80 MB 2025-02-14 06:34:16,935 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -572.45 MB 2025-02-14 06:34:16,935 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39000.74 MB 2025-02-14 06:34:16,935 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46615.49 MB 2025-02-14 06:34:16,935 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7614.76 MB 2025-02-14 06:34:16,935 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43226.10 MB 2025-02-14 06:34:18,859 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 06:34:18,859 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 06:34:18,859 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 06:34:18,859 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:34:18,859 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33992.80 MB 2025-02-14 06:34:18,859 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34523.64 MB 2025-02-14 06:34:18,859 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 06:34:18,859 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46615.49 MB 2025-02-14 06:34:18,859 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38161.87 MB 2025-02-14 06:34:18,859 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8453.62 MB 2025-02-14 06:34:18,859 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38504.01 MB 2025-02-14 06:34:18,873 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 06:34:18,873 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 06:34:18,873 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:34:18,873 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:34:18,873 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34523.64 MB 2025-02-14 06:34:18,873 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36412.86 MB 2025-02-14 06:34:18,873 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.22 MB 2025-02-14 06:34:18,873 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38161.87 MB 2025-02-14 06:34:18,873 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40993.03 MB 2025-02-14 06:34:18,873 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-14 06:34:18,873 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37830.29 MB 2025-02-14 06:34:19,078 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 06:34:19,078 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 06:34:19,078 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 06:34:19,078 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:34:19,078 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36412.86 MB 2025-02-14 06:34:19,078 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38654.71 MB 2025-02-14 06:34:19,078 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 06:34:19,078 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40993.03 MB 2025-02-14 06:34:19,078 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46655.34 MB 2025-02-14 06:34:19,078 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 06:34:19,078 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44199.00 MB 2025-02-14 06:34:19,079 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 06:34:19,079 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 06:34:19,079 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 06:34:19,079 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:34:19,079 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34523.64 MB 2025-02-14 06:34:19,079 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38654.71 MB 2025-02-14 06:34:19,079 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.07 MB 2025-02-14 06:34:19,079 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38161.87 MB 2025-02-14 06:34:19,079 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46655.34 MB 2025-02-14 06:34:19,079 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8493.47 MB 2025-02-14 06:34:19,079 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44199.00 MB 2025-02-14 06:34:19,240 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 06:34:19,240 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 06:34:19,240 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 06:34:19,240 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:34:19,240 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40188.26 MB 2025-02-14 06:34:19,240 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40955.26 MB 2025-02-14 06:34:19,240 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 06:34:19,240 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46655.34 MB 2025-02-14 06:34:19,240 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47072.67 MB 2025-02-14 06:34:19,240 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 06:34:19,240 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41663.05 MB 2025-02-14 06:34:19,258 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 06:34:19,258 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 06:34:19,258 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:34:19,259 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:34:19,259 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41368.15 MB 2025-02-14 06:34:19,259 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41597.61 MB 2025-02-14 06:34:19,259 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.47 MB 2025-02-14 06:34:19,259 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47072.67 MB 2025-02-14 06:34:19,259 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47072.67 MB 2025-02-14 06:34:19,259 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:34:19,259 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41821.57 MB 2025-02-14 06:34:19,260 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 06:34:19,260 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 06:34:19,260 - resource_logging.py:150 - __exit__ - DEBUG - Time: 12.27 seconds 2025-02-14 06:34:19,260 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:34:19,260 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30091.45 MB 2025-02-14 06:34:19,260 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41798.69 MB 2025-02-14 06:34:19,260 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11707.24 MB 2025-02-14 06:34:19,260 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63378.03 MB 2025-02-14 06:34:19,260 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47072.67 MB 2025-02-14 06:34:19,260 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16305.36 MB 2025-02-14 06:34:19,260 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41821.57 MB 2025-02-14 06:34:19,529 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 06:34:19,529 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 06:34:19,529 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 06:34:19,529 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:34:19,529 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41798.69 MB 2025-02-14 06:34:19,529 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35095.84 MB 2025-02-14 06:34:19,529 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6702.85 MB 2025-02-14 06:34:19,529 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47072.67 MB 2025-02-14 06:34:19,529 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47072.67 MB 2025-02-14 06:34:19,529 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:34:19,529 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44310.35 MB 2025-02-14 06:34:19,547 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 06:34:19,547 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 06:34:19,553 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 06:34:19,553 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 06:34:19,553 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:34:19,553 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:34:19,554 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35095.84 MB 2025-02-14 06:34:19,554 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43534.86 MB 2025-02-14 06:34:19,554 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 06:34:19,554 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47072.67 MB 2025-02-14 06:34:19,554 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57562.63 MB 2025-02-14 06:34:19,554 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 06:34:19,554 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43534.86 MB 2025-02-14 06:34:19,705 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 06:34:19,706 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:34:19,706 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 06:34:19,707 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:34:19,707 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 06:34:19,711 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 06:34:19,712 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:34:19,713 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 06:34:19,713 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 06:34:28,240 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:34:28,240 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 06:34:28,245 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 06:34:28,249 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:34:28,249 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 164, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 06:34:28,250 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:34:28,250 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 164, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 06:34:30,828 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 06:34:30,828 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 06:34:30,828 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.57 seconds 2025-02-14 06:34:30,828 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:34:30,829 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29014.87 MB 2025-02-14 06:34:30,829 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29595.26 MB 2025-02-14 06:34:30,829 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 580.39 MB 2025-02-14 06:34:30,829 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 70147.64 MB 2025-02-14 06:34:30,829 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32346.47 MB 2025-02-14 06:34:30,829 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -37801.16 MB 2025-02-14 06:34:30,829 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38487.05 MB 2025-02-14 06:34:30,841 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 06:34:30,841 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 06:34:30,841 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:34:30,841 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:34:30,841 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29595.26 MB 2025-02-14 06:34:30,841 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29856.04 MB 2025-02-14 06:34:30,841 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 260.78 MB 2025-02-14 06:34:30,841 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32346.47 MB 2025-02-14 06:34:30,841 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33487.32 MB 2025-02-14 06:34:30,841 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1140.85 MB 2025-02-14 06:34:30,841 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31872.32 MB 2025-02-14 06:34:31,621 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 06:34:31,621 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 06:34:31,621 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.78 seconds 2025-02-14 06:34:31,621 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:34:31,621 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29856.04 MB 2025-02-14 06:34:31,621 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30069.70 MB 2025-02-14 06:34:31,621 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 213.66 MB 2025-02-14 06:34:31,621 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33487.32 MB 2025-02-14 06:34:31,621 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32445.04 MB 2025-02-14 06:34:31,621 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1042.28 MB 2025-02-14 06:34:31,621 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34027.51 MB 2025-02-14 06:34:31,629 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 06:34:31,629 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 06:34:31,629 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:34:31,629 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:34:31,629 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30069.64 MB 2025-02-14 06:34:31,629 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30829.99 MB 2025-02-14 06:34:31,629 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 760.35 MB 2025-02-14 06:34:31,629 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32445.04 MB 2025-02-14 06:34:31,629 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33210.50 MB 2025-02-14 06:34:31,629 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 765.46 MB 2025-02-14 06:34:31,629 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31400.51 MB 2025-02-14 06:34:31,715 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 06:34:31,715 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 06:34:31,715 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 06:34:31,715 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:34:31,715 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30829.99 MB 2025-02-14 06:34:31,715 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31732.37 MB 2025-02-14 06:34:31,715 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 902.39 MB 2025-02-14 06:34:31,715 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33210.50 MB 2025-02-14 06:34:31,716 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35500.59 MB 2025-02-14 06:34:31,716 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2290.09 MB 2025-02-14 06:34:31,716 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33965.74 MB 2025-02-14 06:34:31,716 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 06:34:31,716 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 06:34:31,716 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 06:34:31,716 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:34:31,716 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30069.64 MB 2025-02-14 06:34:31,716 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31732.37 MB 2025-02-14 06:34:31,716 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1662.74 MB 2025-02-14 06:34:31,716 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32445.04 MB 2025-02-14 06:34:31,716 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35500.59 MB 2025-02-14 06:34:31,716 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3055.55 MB 2025-02-14 06:34:31,716 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33965.74 MB 2025-02-14 06:34:31,782 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 06:34:31,782 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 06:34:31,782 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 06:34:31,782 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:34:31,782 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32349.63 MB 2025-02-14 06:34:31,782 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32660.18 MB 2025-02-14 06:34:31,782 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 310.55 MB 2025-02-14 06:34:31,782 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35500.59 MB 2025-02-14 06:34:31,782 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35668.36 MB 2025-02-14 06:34:31,782 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 167.77 MB 2025-02-14 06:34:31,782 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32957.63 MB 2025-02-14 06:34:31,791 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 06:34:31,791 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 06:34:31,791 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:34:31,791 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:34:31,791 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32826.37 MB 2025-02-14 06:34:31,791 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33053.38 MB 2025-02-14 06:34:31,791 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.01 MB 2025-02-14 06:34:31,791 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35668.36 MB 2025-02-14 06:34:31,791 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35668.36 MB 2025-02-14 06:34:31,791 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:34:31,791 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33076.22 MB 2025-02-14 06:34:31,792 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 06:34:31,792 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 06:34:31,792 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.54 seconds 2025-02-14 06:34:31,792 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:34:31,792 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28443.48 MB 2025-02-14 06:34:31,792 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33254.45 MB 2025-02-14 06:34:31,792 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4810.97 MB 2025-02-14 06:34:31,792 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 70147.64 MB 2025-02-14 06:34:31,792 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35668.36 MB 2025-02-14 06:34:31,792 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34479.28 MB 2025-02-14 06:34:31,792 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33254.45 MB 2025-02-14 06:34:32,063 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 06:34:32,063 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 06:34:32,063 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 06:34:32,063 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:34:32,063 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33254.45 MB 2025-02-14 06:34:32,063 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32321.48 MB 2025-02-14 06:34:32,063 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -932.98 MB 2025-02-14 06:34:32,063 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35668.36 MB 2025-02-14 06:34:32,063 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35668.36 MB 2025-02-14 06:34:32,063 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:34:32,063 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34058.19 MB 2025-02-14 06:34:32,081 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 06:34:32,081 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 06:34:32,087 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 06:34:32,088 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 06:34:32,088 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:34:32,088 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:34:32,088 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32321.48 MB 2025-02-14 06:34:32,088 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40760.50 MB 2025-02-14 06:34:32,088 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 06:34:32,088 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35668.36 MB 2025-02-14 06:34:32,088 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46158.32 MB 2025-02-14 06:34:32,088 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 06:34:32,088 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40760.50 MB 2025-02-14 06:34:32,239 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 06:34:32,240 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:34:32,240 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 06:34:32,241 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:34:32,241 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 06:34:32,246 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 06:34:32,247 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:34:32,247 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 06:34:32,247 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 06:36:05,247 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:36:05,248 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 06:36:05,253 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 06:36:05,258 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:36:05,258 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 165, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 06:36:05,259 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:36:05,259 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 165, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 06:36:07,797 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 06:36:07,797 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 06:36:07,797 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.53 seconds 2025-02-14 06:36:07,797 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:36:07,797 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29021.84 MB 2025-02-14 06:36:07,797 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29605.76 MB 2025-02-14 06:36:07,797 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 583.93 MB 2025-02-14 06:36:07,797 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58743.32 MB 2025-02-14 06:36:07,797 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31344.03 MB 2025-02-14 06:36:07,797 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27399.29 MB 2025-02-14 06:36:07,797 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38494.01 MB 2025-02-14 06:36:07,810 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 06:36:07,810 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 06:36:07,810 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:36:07,810 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:36:07,810 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29605.76 MB 2025-02-14 06:36:07,810 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29861.56 MB 2025-02-14 06:36:07,810 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 255.80 MB 2025-02-14 06:36:07,810 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31344.03 MB 2025-02-14 06:36:07,810 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33332.13 MB 2025-02-14 06:36:07,810 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1988.10 MB 2025-02-14 06:36:07,810 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31887.88 MB 2025-02-14 06:36:08,585 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 06:36:08,585 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 06:36:08,585 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.77 seconds 2025-02-14 06:36:08,585 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:36:08,585 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29861.56 MB 2025-02-14 06:36:08,585 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30075.23 MB 2025-02-14 06:36:08,585 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 213.66 MB 2025-02-14 06:36:08,585 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33332.13 MB 2025-02-14 06:36:08,586 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31232.88 MB 2025-02-14 06:36:08,586 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2099.25 MB 2025-02-14 06:36:08,586 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34034.08 MB 2025-02-14 06:36:08,594 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 06:36:08,594 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 06:36:08,594 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:36:08,594 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:36:08,594 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30075.16 MB 2025-02-14 06:36:08,594 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30835.51 MB 2025-02-14 06:36:08,594 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 760.35 MB 2025-02-14 06:36:08,594 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31232.88 MB 2025-02-14 06:36:08,594 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32761.71 MB 2025-02-14 06:36:08,594 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1528.82 MB 2025-02-14 06:36:08,594 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31406.95 MB 2025-02-14 06:36:08,683 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 06:36:08,683 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 06:36:08,683 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 06:36:08,683 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:36:08,683 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30835.51 MB 2025-02-14 06:36:08,683 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31737.90 MB 2025-02-14 06:36:08,683 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 902.39 MB 2025-02-14 06:36:08,683 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32761.71 MB 2025-02-14 06:36:08,683 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35242.64 MB 2025-02-14 06:36:08,683 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2480.93 MB 2025-02-14 06:36:08,683 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33972.19 MB 2025-02-14 06:36:08,684 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 06:36:08,684 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 06:36:08,684 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 06:36:08,684 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:36:08,684 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30075.16 MB 2025-02-14 06:36:08,684 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31737.90 MB 2025-02-14 06:36:08,684 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1662.74 MB 2025-02-14 06:36:08,684 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31232.88 MB 2025-02-14 06:36:08,684 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35242.64 MB 2025-02-14 06:36:08,684 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4009.75 MB 2025-02-14 06:36:08,684 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33972.19 MB 2025-02-14 06:36:08,749 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 06:36:08,749 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 06:36:08,749 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 06:36:08,749 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:36:08,749 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32355.15 MB 2025-02-14 06:36:08,749 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32666.62 MB 2025-02-14 06:36:08,749 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 311.47 MB 2025-02-14 06:36:08,749 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35242.64 MB 2025-02-14 06:36:08,749 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35410.41 MB 2025-02-14 06:36:08,749 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 167.77 MB 2025-02-14 06:36:08,749 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32962.46 MB 2025-02-14 06:36:08,758 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 06:36:08,758 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 06:36:08,758 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:36:08,759 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:36:08,759 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32832.82 MB 2025-02-14 06:36:08,759 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33057.26 MB 2025-02-14 06:36:08,759 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 224.44 MB 2025-02-14 06:36:08,759 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35410.41 MB 2025-02-14 06:36:08,759 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35410.41 MB 2025-02-14 06:36:08,759 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:36:08,759 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33072.50 MB 2025-02-14 06:36:08,760 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 06:36:08,760 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 06:36:08,760 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.50 seconds 2025-02-14 06:36:08,760 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:36:08,760 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28446.96 MB 2025-02-14 06:36:08,760 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33258.04 MB 2025-02-14 06:36:08,760 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4811.08 MB 2025-02-14 06:36:08,760 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58743.32 MB 2025-02-14 06:36:08,760 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35410.41 MB 2025-02-14 06:36:08,760 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23332.91 MB 2025-02-14 06:36:08,760 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33258.04 MB 2025-02-14 06:36:09,024 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 06:36:09,024 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 06:36:09,024 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 06:36:09,024 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:36:09,024 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33258.04 MB 2025-02-14 06:36:09,024 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32321.31 MB 2025-02-14 06:36:09,024 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -936.73 MB 2025-02-14 06:36:09,024 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35410.41 MB 2025-02-14 06:36:09,024 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35410.41 MB 2025-02-14 06:36:09,024 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:36:09,024 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34060.59 MB 2025-02-14 06:36:09,042 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8150, cut from 8152 2025-02-14 06:36:09,042 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2 ('] 2025-02-14 06:36:09,049 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 06:36:09,049 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 06:36:09,049 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:36:09,049 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:36:09,049 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32321.31 MB 2025-02-14 06:36:09,049 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40747.81 MB 2025-02-14 06:36:09,049 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8426.50 MB 2025-02-14 06:36:09,049 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35410.41 MB 2025-02-14 06:36:09,049 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45883.59 MB 2025-02-14 06:36:09,049 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10473.18 MB 2025-02-14 06:36:09,049 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40747.81 MB 2025-02-14 06:36:09,199 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7942] 2025-02-14 06:36:09,200 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:36:09,200 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 06:36:09,201 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:36:09,201 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 06:36:09,205 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 06:36:09,207 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:36:09,207 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 06:36:09,207 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2 ('] 2025-02-14 06:37:18,058 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:37:18,058 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 06:37:18,062 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 06:37:18,066 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:37:18,066 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2222, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 06:37:18,067 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:37:18,067 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2222, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 06:37:52,265 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 06:37:52,265 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 06:37:52,265 - resource_logging.py:150 - __exit__ - DEBUG - Time: 34.19 seconds 2025-02-14 06:37:52,265 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:37:52,265 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43355.34 MB 2025-02-14 06:37:52,265 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 51219.66 MB 2025-02-14 06:37:52,265 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7864.32 MB 2025-02-14 06:37:52,265 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58449.72 MB 2025-02-14 06:37:52,265 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53661.93 MB 2025-02-14 06:37:52,265 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4787.80 MB 2025-02-14 06:37:52,265 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 60075.28 MB 2025-02-14 06:37:52,414 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 06:37:52,414 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 06:37:52,414 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 06:37:52,414 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:37:52,414 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 51219.66 MB 2025-02-14 06:37:52,414 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42233.77 MB 2025-02-14 06:37:52,414 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8985.89 MB 2025-02-14 06:37:52,414 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53661.93 MB 2025-02-14 06:37:52,414 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 87216.36 MB 2025-02-14 06:37:52,414 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 33554.43 MB 2025-02-14 06:37:52,414 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 74140.39 MB 2025-02-14 06:37:54,383 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 06:37:54,383 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 06:37:54,383 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.97 seconds 2025-02-14 06:37:54,383 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:37:54,383 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42233.77 MB 2025-02-14 06:37:54,384 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42764.61 MB 2025-02-14 06:37:54,384 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 06:37:54,384 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 87216.36 MB 2025-02-14 06:37:54,384 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44975.52 MB 2025-02-14 06:37:54,384 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -42240.84 MB 2025-02-14 06:37:54,384 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46746.02 MB 2025-02-14 06:37:54,398 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 06:37:54,398 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 06:37:54,398 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:37:54,398 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:37:54,398 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42764.61 MB 2025-02-14 06:37:54,398 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44653.83 MB 2025-02-14 06:37:54,398 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.22 MB 2025-02-14 06:37:54,398 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44975.52 MB 2025-02-14 06:37:54,398 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48752.49 MB 2025-02-14 06:37:54,398 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3776.97 MB 2025-02-14 06:37:54,398 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46071.26 MB 2025-02-14 06:37:54,602 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 06:37:54,602 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 06:37:54,602 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 06:37:54,602 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:37:54,602 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44653.83 MB 2025-02-14 06:37:54,602 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46895.68 MB 2025-02-14 06:37:54,603 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 06:37:54,603 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48752.49 MB 2025-02-14 06:37:54,603 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54886.66 MB 2025-02-14 06:37:54,603 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 06:37:54,603 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52439.97 MB 2025-02-14 06:37:54,603 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 06:37:54,603 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 06:37:54,603 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 06:37:54,603 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:37:54,603 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42764.61 MB 2025-02-14 06:37:54,603 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46895.68 MB 2025-02-14 06:37:54,603 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.07 MB 2025-02-14 06:37:54,603 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44975.52 MB 2025-02-14 06:37:54,603 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54886.66 MB 2025-02-14 06:37:54,603 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9911.14 MB 2025-02-14 06:37:54,603 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52439.97 MB 2025-02-14 06:37:54,766 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 06:37:54,766 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 06:37:54,766 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 06:37:54,766 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:37:54,766 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 48429.23 MB 2025-02-14 06:37:54,766 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 49196.23 MB 2025-02-14 06:37:54,766 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 06:37:54,766 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54886.66 MB 2025-02-14 06:37:54,766 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55306.09 MB 2025-02-14 06:37:54,766 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 419.43 MB 2025-02-14 06:37:54,766 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49904.02 MB 2025-02-14 06:37:54,784 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 06:37:54,784 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 06:37:54,784 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:37:54,785 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:37:54,785 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 49609.12 MB 2025-02-14 06:37:54,785 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 49837.50 MB 2025-02-14 06:37:54,785 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.38 MB 2025-02-14 06:37:54,785 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55306.09 MB 2025-02-14 06:37:54,785 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55306.09 MB 2025-02-14 06:37:54,785 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:37:54,785 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50051.09 MB 2025-02-14 06:37:54,786 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 06:37:54,786 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 06:37:54,786 - resource_logging.py:150 - __exit__ - DEBUG - Time: 36.72 seconds 2025-02-14 06:37:54,786 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:37:54,786 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35613.72 MB 2025-02-14 06:37:54,786 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 50038.57 MB 2025-02-14 06:37:54,786 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14424.85 MB 2025-02-14 06:37:54,786 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58449.72 MB 2025-02-14 06:37:54,786 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55306.09 MB 2025-02-14 06:37:54,786 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3143.63 MB 2025-02-14 06:37:54,786 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50051.09 MB 2025-02-14 06:37:55,054 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 06:37:55,054 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 06:37:55,054 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 06:37:55,054 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:37:55,054 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 50038.57 MB 2025-02-14 06:37:55,054 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40617.84 MB 2025-02-14 06:37:55,054 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9420.73 MB 2025-02-14 06:37:55,054 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55306.09 MB 2025-02-14 06:37:55,054 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55306.09 MB 2025-02-14 06:37:55,054 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:37:55,054 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52550.24 MB 2025-02-14 06:37:55,072 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 06:37:55,072 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 06:37:55,078 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 06:37:55,079 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 06:37:55,079 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:37:55,079 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:37:55,079 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40617.84 MB 2025-02-14 06:37:55,079 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 49056.87 MB 2025-02-14 06:37:55,079 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 06:37:55,079 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55306.09 MB 2025-02-14 06:37:55,079 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 63696.80 MB 2025-02-14 06:37:55,079 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 06:37:55,079 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49056.87 MB 2025-02-14 06:37:55,233 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 06:37:55,234 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:37:55,234 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 06:37:55,235 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:37:55,235 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 06:37:55,240 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 06:37:55,241 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:37:55,241 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 06:37:55,241 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 06:39:10,272 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:39:10,272 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 06:39:10,277 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 06:39:10,281 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:39:10,281 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1572, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 06:39:10,282 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:39:10,282 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1572, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 06:39:34,544 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 06:39:34,544 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 06:39:34,544 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.25 seconds 2025-02-14 06:39:34,544 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:39:34,544 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23922.65 MB 2025-02-14 06:39:34,544 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29486.33 MB 2025-02-14 06:39:34,544 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5563.68 MB 2025-02-14 06:39:34,544 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 76281.81 MB 2025-02-14 06:39:34,544 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39233.52 MB 2025-02-14 06:39:34,544 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -37048.29 MB 2025-02-14 06:39:34,545 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38377.60 MB 2025-02-14 06:39:34,636 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 06:39:34,636 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 06:39:34,636 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 06:39:34,636 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:39:34,636 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29486.33 MB 2025-02-14 06:39:34,636 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23950.19 MB 2025-02-14 06:39:34,636 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5536.14 MB 2025-02-14 06:39:34,636 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39233.52 MB 2025-02-14 06:39:34,636 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51810.14 MB 2025-02-14 06:39:34,636 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 12576.62 MB 2025-02-14 06:39:34,636 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46074.40 MB 2025-02-14 06:39:36,553 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 06:39:36,553 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 06:39:36,554 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 06:39:36,554 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:39:36,554 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23950.19 MB 2025-02-14 06:39:36,554 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24481.03 MB 2025-02-14 06:39:36,554 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 06:39:36,554 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51810.14 MB 2025-02-14 06:39:36,554 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29481.76 MB 2025-02-14 06:39:36,554 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22328.38 MB 2025-02-14 06:39:36,554 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28460.36 MB 2025-02-14 06:39:36,570 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 06:39:36,570 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 06:39:36,570 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:39:36,570 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:39:36,570 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24481.03 MB 2025-02-14 06:39:36,570 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26370.56 MB 2025-02-14 06:39:36,570 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 06:39:36,570 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29481.76 MB 2025-02-14 06:39:36,570 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30425.48 MB 2025-02-14 06:39:36,570 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 06:39:36,570 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27787.99 MB 2025-02-14 06:39:36,775 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 06:39:36,775 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 06:39:36,775 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 06:39:36,775 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:39:36,775 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26370.56 MB 2025-02-14 06:39:36,775 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28612.42 MB 2025-02-14 06:39:36,775 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 06:39:36,775 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30425.48 MB 2025-02-14 06:39:36,775 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36559.65 MB 2025-02-14 06:39:36,775 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 06:39:36,775 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34156.70 MB 2025-02-14 06:39:36,776 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 06:39:36,776 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 06:39:36,776 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 06:39:36,776 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:39:36,776 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24481.03 MB 2025-02-14 06:39:36,776 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28612.42 MB 2025-02-14 06:39:36,776 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 06:39:36,776 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29481.76 MB 2025-02-14 06:39:36,776 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36559.65 MB 2025-02-14 06:39:36,776 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7077.89 MB 2025-02-14 06:39:36,776 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34156.70 MB 2025-02-14 06:39:36,937 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 06:39:36,937 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 06:39:36,937 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 06:39:36,937 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:39:36,937 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30145.96 MB 2025-02-14 06:39:36,937 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30912.96 MB 2025-02-14 06:39:36,937 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 06:39:36,937 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36559.65 MB 2025-02-14 06:39:36,937 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36974.89 MB 2025-02-14 06:39:36,937 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 06:39:36,937 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31620.75 MB 2025-02-14 06:39:36,956 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 06:39:36,956 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 06:39:36,956 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:39:36,956 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:39:36,956 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31325.85 MB 2025-02-14 06:39:36,956 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31553.67 MB 2025-02-14 06:39:36,956 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.82 MB 2025-02-14 06:39:36,956 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36974.89 MB 2025-02-14 06:39:36,956 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36974.89 MB 2025-02-14 06:39:36,956 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:39:36,956 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31792.21 MB 2025-02-14 06:39:36,958 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 06:39:36,958 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 06:39:36,958 - resource_logging.py:150 - __exit__ - DEBUG - Time: 26.67 seconds 2025-02-14 06:39:36,958 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:39:36,958 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18445.68 MB 2025-02-14 06:39:36,958 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31754.57 MB 2025-02-14 06:39:36,958 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13308.89 MB 2025-02-14 06:39:36,958 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 76281.81 MB 2025-02-14 06:39:36,958 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36974.89 MB 2025-02-14 06:39:36,958 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -39306.92 MB 2025-02-14 06:39:36,958 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31792.21 MB 2025-02-14 06:39:37,227 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 06:39:37,228 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 06:39:37,228 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 06:39:37,228 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:39:37,228 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20435.95 MB 2025-02-14 06:39:37,228 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23447.40 MB 2025-02-14 06:39:37,228 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3011.45 MB 2025-02-14 06:39:37,228 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36974.89 MB 2025-02-14 06:39:37,228 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36974.89 MB 2025-02-14 06:39:37,228 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:39:37,228 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23748.51 MB 2025-02-14 06:39:37,246 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8155, cut from 8157 2025-02-14 06:39:37,246 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 06:39:37,253 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 06:39:37,253 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 06:39:37,253 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:39:37,253 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:39:37,253 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23447.40 MB 2025-02-14 06:39:37,253 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31878.87 MB 2025-02-14 06:39:37,253 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8431.46 MB 2025-02-14 06:39:37,253 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36974.89 MB 2025-02-14 06:39:37,253 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45359.30 MB 2025-02-14 06:39:37,253 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8384.41 MB 2025-02-14 06:39:37,253 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31878.87 MB 2025-02-14 06:39:37,405 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7947] 2025-02-14 06:39:37,406 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:39:37,406 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 06:39:37,407 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:39:37,407 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 06:39:37,412 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 06:39:37,413 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:39:37,413 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 06:39:37,413 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 06:39:47,133 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:39:47,133 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 06:39:47,138 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 06:39:47,142 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:39:47,142 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2079, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 06:39:47,143 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:39:47,143 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2079, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 06:40:19,660 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 06:40:19,660 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 06:40:19,660 - resource_logging.py:150 - __exit__ - DEBUG - Time: 32.51 seconds 2025-02-14 06:40:19,660 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:40:19,660 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27455.51 MB 2025-02-14 06:40:19,660 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34812.98 MB 2025-02-14 06:40:19,661 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7357.46 MB 2025-02-14 06:40:19,661 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53743.71 MB 2025-02-14 06:40:19,661 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41030.78 MB 2025-02-14 06:40:19,661 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12712.94 MB 2025-02-14 06:40:19,661 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43722.46 MB 2025-02-14 06:40:19,758 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 06:40:19,758 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 06:40:19,758 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 06:40:19,758 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:40:19,758 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34812.98 MB 2025-02-14 06:40:19,758 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26585.92 MB 2025-02-14 06:40:19,759 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8227.05 MB 2025-02-14 06:40:19,759 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41030.78 MB 2025-02-14 06:40:19,759 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53791.95 MB 2025-02-14 06:40:19,759 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 12761.17 MB 2025-02-14 06:40:19,759 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47041.94 MB 2025-02-14 06:40:21,701 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 06:40:21,701 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 06:40:21,701 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-14 06:40:21,701 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:40:21,702 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26585.92 MB 2025-02-14 06:40:21,702 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27116.77 MB 2025-02-14 06:40:21,702 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 06:40:21,702 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53791.95 MB 2025-02-14 06:40:21,702 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30897.34 MB 2025-02-14 06:40:21,702 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22894.61 MB 2025-02-14 06:40:21,702 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31097.14 MB 2025-02-14 06:40:21,715 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 06:40:21,716 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 06:40:21,716 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:40:21,716 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:40:21,716 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27116.77 MB 2025-02-14 06:40:21,716 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29006.30 MB 2025-02-14 06:40:21,716 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 06:40:21,716 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30897.34 MB 2025-02-14 06:40:21,716 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33728.50 MB 2025-02-14 06:40:21,716 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-14 06:40:21,716 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30423.73 MB 2025-02-14 06:40:21,928 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 06:40:21,928 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 06:40:21,929 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 06:40:21,929 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:40:21,929 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29006.30 MB 2025-02-14 06:40:21,929 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31248.16 MB 2025-02-14 06:40:21,929 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 06:40:21,929 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33728.50 MB 2025-02-14 06:40:21,929 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39390.81 MB 2025-02-14 06:40:21,929 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 06:40:21,929 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36792.44 MB 2025-02-14 06:40:21,929 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 06:40:21,929 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 06:40:21,929 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 06:40:21,929 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:40:21,929 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27116.77 MB 2025-02-14 06:40:21,929 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31248.16 MB 2025-02-14 06:40:21,929 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 06:40:21,929 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30897.34 MB 2025-02-14 06:40:21,929 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39390.81 MB 2025-02-14 06:40:21,929 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8493.47 MB 2025-02-14 06:40:21,929 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36792.44 MB 2025-02-14 06:40:22,095 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 06:40:22,095 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 06:40:22,095 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 06:40:22,095 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:40:22,096 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32781.70 MB 2025-02-14 06:40:22,096 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33548.70 MB 2025-02-14 06:40:22,096 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 06:40:22,096 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39390.81 MB 2025-02-14 06:40:22,096 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39806.04 MB 2025-02-14 06:40:22,096 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 06:40:22,096 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34256.49 MB 2025-02-14 06:40:22,115 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 06:40:22,115 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 06:40:22,115 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:40:22,115 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:40:22,115 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33961.59 MB 2025-02-14 06:40:22,115 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34189.27 MB 2025-02-14 06:40:22,115 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.68 MB 2025-02-14 06:40:22,115 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39806.04 MB 2025-02-14 06:40:22,115 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39806.04 MB 2025-02-14 06:40:22,115 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:40:22,115 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34413.09 MB 2025-02-14 06:40:22,116 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 06:40:22,116 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 06:40:22,116 - resource_logging.py:150 - __exit__ - DEBUG - Time: 34.97 seconds 2025-02-14 06:40:22,116 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:40:22,116 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20212.11 MB 2025-02-14 06:40:22,116 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34389.98 MB 2025-02-14 06:40:22,116 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14177.87 MB 2025-02-14 06:40:22,116 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53743.71 MB 2025-02-14 06:40:22,116 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39806.04 MB 2025-02-14 06:40:22,116 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13937.67 MB 2025-02-14 06:40:22,116 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34413.09 MB 2025-02-14 06:40:22,389 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 06:40:22,389 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 06:40:22,389 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 06:40:22,389 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:40:22,389 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34389.98 MB 2025-02-14 06:40:22,389 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25210.78 MB 2025-02-14 06:40:22,389 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9179.19 MB 2025-02-14 06:40:22,389 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39806.04 MB 2025-02-14 06:40:22,389 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39806.04 MB 2025-02-14 06:40:22,389 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:40:22,389 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36897.03 MB 2025-02-14 06:40:22,407 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8147, cut from 8149 2025-02-14 06:40:22,407 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 06:40:22,413 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 06:40:22,413 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 06:40:22,413 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:40:22,413 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:40:22,413 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25210.78 MB 2025-02-14 06:40:22,413 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33633.99 MB 2025-02-14 06:40:22,413 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8423.21 MB 2025-02-14 06:40:22,413 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39806.04 MB 2025-02-14 06:40:22,413 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48182.07 MB 2025-02-14 06:40:22,413 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8376.03 MB 2025-02-14 06:40:22,413 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33633.99 MB 2025-02-14 06:40:22,575 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7939] 2025-02-14 06:40:22,577 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:40:22,577 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 06:40:22,578 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:40:22,578 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 06:40:22,582 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 06:40:22,584 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:40:22,584 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 06:40:22,584 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 06:40:33,033 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:40:33,033 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 06:40:33,038 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 06:40:33,042 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:40:33,042 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 174, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 06:40:33,043 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:40:33,043 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 174, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 06:40:35,783 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 06:40:35,783 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 06:40:35,783 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.74 seconds 2025-02-14 06:40:35,783 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:40:35,783 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14181.17 MB 2025-02-14 06:40:35,783 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14796.94 MB 2025-02-14 06:40:35,783 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 615.78 MB 2025-02-14 06:40:35,783 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56558.09 MB 2025-02-14 06:40:35,783 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 16907.24 MB 2025-02-14 06:40:35,783 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -39650.85 MB 2025-02-14 06:40:35,783 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23653.34 MB 2025-02-14 06:40:35,797 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 06:40:35,797 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 06:40:35,797 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:40:35,797 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:40:35,797 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14796.94 MB 2025-02-14 06:40:35,797 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15005.41 MB 2025-02-14 06:40:35,797 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 208.47 MB 2025-02-14 06:40:35,797 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 16907.24 MB 2025-02-14 06:40:35,797 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18324.91 MB 2025-02-14 06:40:35,797 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1417.67 MB 2025-02-14 06:40:35,797 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17110.05 MB 2025-02-14 06:40:36,583 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 06:40:36,583 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 06:40:36,583 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.78 seconds 2025-02-14 06:40:36,583 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:40:36,583 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15005.41 MB 2025-02-14 06:40:36,583 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15219.08 MB 2025-02-14 06:40:36,583 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 213.66 MB 2025-02-14 06:40:36,583 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18324.91 MB 2025-02-14 06:40:36,583 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17754.49 MB 2025-02-14 06:40:36,583 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -570.43 MB 2025-02-14 06:40:36,583 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19176.89 MB 2025-02-14 06:40:36,591 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 06:40:36,591 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 06:40:36,591 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 06:40:36,591 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:40:36,591 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15219.01 MB 2025-02-14 06:40:36,591 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15979.36 MB 2025-02-14 06:40:36,591 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 760.35 MB 2025-02-14 06:40:36,591 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17754.49 MB 2025-02-14 06:40:36,591 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17754.49 MB 2025-02-14 06:40:36,591 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:40:36,591 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16549.88 MB 2025-02-14 06:40:36,683 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 06:40:36,683 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 06:40:36,683 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 06:40:36,683 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:40:36,683 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15979.36 MB 2025-02-14 06:40:36,683 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16881.75 MB 2025-02-14 06:40:36,683 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 902.39 MB 2025-02-14 06:40:36,683 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17754.49 MB 2025-02-14 06:40:36,683 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20235.42 MB 2025-02-14 06:40:36,683 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2480.93 MB 2025-02-14 06:40:36,683 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19116.04 MB 2025-02-14 06:40:36,683 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 06:40:36,683 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 06:40:36,683 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 06:40:36,683 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:40:36,684 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15219.01 MB 2025-02-14 06:40:36,684 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16881.75 MB 2025-02-14 06:40:36,684 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1662.74 MB 2025-02-14 06:40:36,684 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17754.49 MB 2025-02-14 06:40:36,684 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20235.42 MB 2025-02-14 06:40:36,684 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2480.93 MB 2025-02-14 06:40:36,684 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19116.04 MB 2025-02-14 06:40:36,752 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 06:40:36,752 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 06:40:36,752 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 06:40:36,752 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:40:36,752 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17499.00 MB 2025-02-14 06:40:36,752 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17809.55 MB 2025-02-14 06:40:36,752 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 310.55 MB 2025-02-14 06:40:36,752 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20235.42 MB 2025-02-14 06:40:36,752 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20401.09 MB 2025-02-14 06:40:36,752 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 165.68 MB 2025-02-14 06:40:36,752 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18103.19 MB 2025-02-14 06:40:36,761 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 06:40:36,761 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 06:40:36,761 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:40:36,761 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:40:36,761 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17975.75 MB 2025-02-14 06:40:36,761 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18204.03 MB 2025-02-14 06:40:36,761 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.28 MB 2025-02-14 06:40:36,761 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20401.09 MB 2025-02-14 06:40:36,761 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20401.09 MB 2025-02-14 06:40:36,761 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:40:36,761 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18225.57 MB 2025-02-14 06:40:36,762 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 06:40:36,762 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 06:40:36,762 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.72 seconds 2025-02-14 06:40:36,762 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:40:36,762 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13574.94 MB 2025-02-14 06:40:36,762 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18405.11 MB 2025-02-14 06:40:36,762 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4830.17 MB 2025-02-14 06:40:36,762 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56558.09 MB 2025-02-14 06:40:36,762 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20401.09 MB 2025-02-14 06:40:36,762 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -36157.00 MB 2025-02-14 06:40:36,762 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18405.11 MB 2025-02-14 06:40:37,030 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 06:40:37,030 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 06:40:37,030 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 06:40:37,031 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:40:37,031 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18405.11 MB 2025-02-14 06:40:37,031 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17453.25 MB 2025-02-14 06:40:37,031 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -951.86 MB 2025-02-14 06:40:37,031 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20401.09 MB 2025-02-14 06:40:37,031 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20401.09 MB 2025-02-14 06:40:37,031 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:40:37,031 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19208.84 MB 2025-02-14 06:40:37,049 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 06:40:37,049 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 06:40:37,055 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 06:40:37,055 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 06:40:37,055 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:40:37,055 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:40:37,055 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17453.25 MB 2025-02-14 06:40:37,055 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25892.27 MB 2025-02-14 06:40:37,055 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 06:40:37,055 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20401.09 MB 2025-02-14 06:40:37,055 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30891.05 MB 2025-02-14 06:40:37,055 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 06:40:37,055 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25892.27 MB 2025-02-14 06:40:37,218 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 06:40:37,219 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:40:37,219 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 06:40:37,220 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:40:37,220 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 06:40:37,225 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 06:40:37,226 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:40:37,226 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 06:40:37,226 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 06:41:42,964 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:41:42,964 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 06:41:42,969 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 06:41:42,973 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:41:42,973 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 188, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 06:41:42,974 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:41:42,974 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 188, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 06:41:45,860 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 06:41:45,860 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 06:41:45,861 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.88 seconds 2025-02-14 06:41:45,861 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:41:45,861 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14278.72 MB 2025-02-14 06:41:45,861 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14944.04 MB 2025-02-14 06:41:45,861 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 665.32 MB 2025-02-14 06:41:45,861 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43476.06 MB 2025-02-14 06:41:45,861 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18433.97 MB 2025-02-14 06:41:45,861 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25042.09 MB 2025-02-14 06:41:45,861 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23750.90 MB 2025-02-14 06:41:45,875 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 06:41:45,875 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 06:41:45,875 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:41:45,876 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:41:45,876 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14944.04 MB 2025-02-14 06:41:45,876 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15266.39 MB 2025-02-14 06:41:45,876 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 322.35 MB 2025-02-14 06:41:45,876 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18433.97 MB 2025-02-14 06:41:45,876 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19100.86 MB 2025-02-14 06:41:45,876 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 666.89 MB 2025-02-14 06:41:45,876 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17627.23 MB 2025-02-14 06:41:46,775 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 06:41:46,775 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 06:41:46,775 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.90 seconds 2025-02-14 06:41:46,775 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:41:46,775 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15266.39 MB 2025-02-14 06:41:46,775 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15515.88 MB 2025-02-14 06:41:46,775 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 249.50 MB 2025-02-14 06:41:46,775 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19100.86 MB 2025-02-14 06:41:46,775 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18719.18 MB 2025-02-14 06:41:46,775 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -381.68 MB 2025-02-14 06:41:46,775 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19437.86 MB 2025-02-14 06:41:46,783 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 06:41:46,783 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 06:41:46,783 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:41:46,783 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:41:46,783 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15515.82 MB 2025-02-14 06:41:46,783 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16403.68 MB 2025-02-14 06:41:46,783 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 887.87 MB 2025-02-14 06:41:46,783 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18719.18 MB 2025-02-14 06:41:46,783 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18719.18 MB 2025-02-14 06:41:46,783 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:41:46,783 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17069.88 MB 2025-02-14 06:41:46,884 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 06:41:46,884 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 06:41:46,884 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 06:41:46,884 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:41:46,884 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16403.68 MB 2025-02-14 06:41:46,884 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17457.39 MB 2025-02-14 06:41:46,884 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1053.71 MB 2025-02-14 06:41:46,884 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18719.18 MB 2025-02-14 06:41:46,884 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21164.46 MB 2025-02-14 06:41:46,884 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2445.28 MB 2025-02-14 06:41:46,884 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20067.89 MB 2025-02-14 06:41:46,885 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 06:41:46,885 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 06:41:46,885 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 06:41:46,885 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:41:46,885 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15515.82 MB 2025-02-14 06:41:46,885 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17457.39 MB 2025-02-14 06:41:46,885 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1941.57 MB 2025-02-14 06:41:46,885 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18719.18 MB 2025-02-14 06:41:46,885 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21164.46 MB 2025-02-14 06:41:46,885 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2445.28 MB 2025-02-14 06:41:46,885 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20067.89 MB 2025-02-14 06:41:46,961 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 06:41:46,961 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 06:41:46,961 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 06:41:46,961 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:41:46,961 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18178.16 MB 2025-02-14 06:41:46,961 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18539.17 MB 2025-02-14 06:41:46,961 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 361.02 MB 2025-02-14 06:41:46,961 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21164.46 MB 2025-02-14 06:41:46,961 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21359.49 MB 2025-02-14 06:41:46,961 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 195.04 MB 2025-02-14 06:41:46,961 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18877.15 MB 2025-02-14 06:41:46,971 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 06:41:46,971 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 06:41:46,971 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:41:46,971 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:41:46,972 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18733.24 MB 2025-02-14 06:41:46,972 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18945.14 MB 2025-02-14 06:41:46,972 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 211.91 MB 2025-02-14 06:41:46,972 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21359.49 MB 2025-02-14 06:41:46,972 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21359.49 MB 2025-02-14 06:41:46,972 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:41:46,972 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18990.03 MB 2025-02-14 06:41:46,973 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 06:41:46,973 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 06:41:46,973 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.00 seconds 2025-02-14 06:41:46,973 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:41:46,973 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13623.71 MB 2025-02-14 06:41:46,973 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19146.22 MB 2025-02-14 06:41:46,973 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5522.50 MB 2025-02-14 06:41:46,973 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43476.06 MB 2025-02-14 06:41:46,973 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21359.49 MB 2025-02-14 06:41:46,973 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22116.56 MB 2025-02-14 06:41:46,973 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19146.22 MB 2025-02-14 06:41:47,238 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 06:41:47,238 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 06:41:47,238 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 06:41:47,238 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:41:47,238 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19146.22 MB 2025-02-14 06:41:47,238 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17627.61 MB 2025-02-14 06:41:47,238 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1518.60 MB 2025-02-14 06:41:47,238 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21359.49 MB 2025-02-14 06:41:47,239 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21359.49 MB 2025-02-14 06:41:47,239 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:41:47,239 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19146.22 MB 2025-02-14 06:41:47,256 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 06:41:47,257 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 06:41:47,263 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 06:41:47,263 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 06:41:47,263 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:41:47,263 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:41:47,263 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17627.61 MB 2025-02-14 06:41:47,263 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26066.64 MB 2025-02-14 06:41:47,263 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 06:41:47,263 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21359.49 MB 2025-02-14 06:41:47,263 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31849.45 MB 2025-02-14 06:41:47,263 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 06:41:47,263 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26066.64 MB 2025-02-14 06:41:47,413 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 06:41:47,414 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:41:47,414 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 06:41:47,415 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:41:47,415 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 06:41:47,420 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 06:41:47,421 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:41:47,421 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 06:41:47,421 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 06:43:18,282 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:43:18,283 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 06:43:18,287 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 06:43:18,291 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:43:18,291 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1656, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 06:43:18,292 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:43:18,292 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1656, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 06:43:43,694 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 06:43:43,694 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 06:43:43,694 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.39 seconds 2025-02-14 06:43:43,694 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:43:43,694 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24507.98 MB 2025-02-14 06:43:43,694 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30369.52 MB 2025-02-14 06:43:43,694 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5861.54 MB 2025-02-14 06:43:43,694 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44434.46 MB 2025-02-14 06:43:43,694 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39548.09 MB 2025-02-14 06:43:43,694 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4886.36 MB 2025-02-14 06:43:43,694 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39189.48 MB 2025-02-14 06:43:43,791 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 06:43:43,791 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 06:43:43,791 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 06:43:43,792 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:43:43,792 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30369.52 MB 2025-02-14 06:43:43,792 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24386.88 MB 2025-02-14 06:43:43,792 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5982.64 MB 2025-02-14 06:43:43,792 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39548.09 MB 2025-02-14 06:43:43,792 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55683.58 MB 2025-02-14 06:43:43,792 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 16135.49 MB 2025-02-14 06:43:43,792 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47099.34 MB 2025-02-14 06:43:45,712 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 06:43:45,712 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 06:43:45,712 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 06:43:45,712 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:43:45,712 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24386.88 MB 2025-02-14 06:43:45,712 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24917.72 MB 2025-02-14 06:43:45,712 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 06:43:45,712 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55683.58 MB 2025-02-14 06:43:45,712 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30907.83 MB 2025-02-14 06:43:45,712 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24775.75 MB 2025-02-14 06:43:45,712 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28897.05 MB 2025-02-14 06:43:45,729 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 06:43:45,729 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 06:43:45,729 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:43:45,729 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:43:45,729 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24917.72 MB 2025-02-14 06:43:45,729 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26807.25 MB 2025-02-14 06:43:45,729 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 06:43:45,729 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30907.83 MB 2025-02-14 06:43:45,729 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31851.54 MB 2025-02-14 06:43:45,729 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 06:43:45,729 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28224.68 MB 2025-02-14 06:43:45,942 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 06:43:45,942 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 06:43:45,942 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 06:43:45,943 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:43:45,943 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26807.25 MB 2025-02-14 06:43:45,943 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29049.11 MB 2025-02-14 06:43:45,943 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 06:43:45,943 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31851.54 MB 2025-02-14 06:43:45,943 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37513.85 MB 2025-02-14 06:43:45,943 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 06:43:45,943 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34593.39 MB 2025-02-14 06:43:45,943 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 06:43:45,943 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 06:43:45,943 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 06:43:45,943 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:43:45,943 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24917.72 MB 2025-02-14 06:43:45,943 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29049.11 MB 2025-02-14 06:43:45,943 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 06:43:45,943 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30907.83 MB 2025-02-14 06:43:45,943 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37513.85 MB 2025-02-14 06:43:45,943 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 06:43:45,943 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34593.39 MB 2025-02-14 06:43:46,113 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 06:43:46,113 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 06:43:46,113 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 06:43:46,113 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:43:46,113 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30582.65 MB 2025-02-14 06:43:46,113 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31349.65 MB 2025-02-14 06:43:46,113 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 06:43:46,113 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37513.85 MB 2025-02-14 06:43:46,113 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37931.19 MB 2025-02-14 06:43:46,113 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 06:43:46,113 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32057.44 MB 2025-02-14 06:43:46,133 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 06:43:46,133 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 06:43:46,133 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:43:46,133 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:43:46,133 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31762.54 MB 2025-02-14 06:43:46,133 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31988.87 MB 2025-02-14 06:43:46,133 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.33 MB 2025-02-14 06:43:46,133 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37931.19 MB 2025-02-14 06:43:46,133 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37931.19 MB 2025-02-14 06:43:46,133 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:43:46,133 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32179.40 MB 2025-02-14 06:43:46,134 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 06:43:46,134 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 06:43:46,134 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.84 seconds 2025-02-14 06:43:46,134 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:43:46,134 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18738.34 MB 2025-02-14 06:43:46,134 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32189.55 MB 2025-02-14 06:43:46,134 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13451.21 MB 2025-02-14 06:43:46,134 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44434.46 MB 2025-02-14 06:43:46,134 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37931.19 MB 2025-02-14 06:43:46,134 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6503.27 MB 2025-02-14 06:43:46,134 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32189.55 MB 2025-02-14 06:43:46,403 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 06:43:46,403 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 06:43:46,403 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 06:43:46,403 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:43:46,403 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32189.55 MB 2025-02-14 06:43:46,403 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23736.64 MB 2025-02-14 06:43:46,403 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8452.92 MB 2025-02-14 06:43:46,403 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37931.19 MB 2025-02-14 06:43:46,403 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37931.19 MB 2025-02-14 06:43:46,403 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:43:46,403 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34696.31 MB 2025-02-14 06:43:46,421 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8146, cut from 8148 2025-02-14 06:43:46,421 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 06:43:46,427 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 06:43:46,427 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 06:43:46,427 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:43:46,427 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:43:46,427 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23736.64 MB 2025-02-14 06:43:46,427 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32158.97 MB 2025-02-14 06:43:46,427 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8422.33 MB 2025-02-14 06:43:46,427 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37931.19 MB 2025-02-14 06:43:46,427 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46305.12 MB 2025-02-14 06:43:46,428 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8373.93 MB 2025-02-14 06:43:46,428 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32158.97 MB 2025-02-14 06:43:46,594 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7938] 2025-02-14 06:43:46,595 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:43:46,595 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 06:43:46,596 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:43:46,596 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 06:43:46,601 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 06:43:46,602 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:43:46,602 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 06:43:46,602 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 06:44:44,043 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:44:44,043 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 06:44:44,048 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 06:44:44,052 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:44:44,052 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2114, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 06:44:44,053 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:44:44,053 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2114, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 06:45:16,898 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 06:45:16,898 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 06:45:16,898 - resource_logging.py:150 - __exit__ - DEBUG - Time: 32.84 seconds 2025-02-14 06:45:16,898 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:45:16,898 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27699.40 MB 2025-02-14 06:45:16,898 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35180.72 MB 2025-02-14 06:45:16,898 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7481.33 MB 2025-02-14 06:45:16,899 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58864.96 MB 2025-02-14 06:45:16,899 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41137.73 MB 2025-02-14 06:45:16,899 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17727.23 MB 2025-02-14 06:45:16,899 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44192.84 MB 2025-02-14 06:45:17,033 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 06:45:17,033 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 06:45:17,033 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 06:45:17,033 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:45:17,033 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35180.72 MB 2025-02-14 06:45:17,033 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26767.88 MB 2025-02-14 06:45:17,033 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8412.85 MB 2025-02-14 06:45:17,033 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41137.73 MB 2025-02-14 06:45:17,033 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65760.40 MB 2025-02-14 06:45:17,033 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 24622.66 MB 2025-02-14 06:45:17,033 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 55779.70 MB 2025-02-14 06:45:18,982 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 06:45:18,982 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 06:45:18,982 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.95 seconds 2025-02-14 06:45:18,982 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:45:18,982 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26767.88 MB 2025-02-14 06:45:18,982 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27298.72 MB 2025-02-14 06:45:18,982 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 06:45:18,982 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65760.40 MB 2025-02-14 06:45:18,982 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30884.76 MB 2025-02-14 06:45:18,982 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34875.64 MB 2025-02-14 06:45:18,982 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31279.09 MB 2025-02-14 06:45:18,996 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 06:45:18,996 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 06:45:18,996 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:45:18,996 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:45:18,996 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27298.72 MB 2025-02-14 06:45:18,996 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29188.25 MB 2025-02-14 06:45:18,996 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 06:45:18,996 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30884.76 MB 2025-02-14 06:45:18,996 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33715.91 MB 2025-02-14 06:45:18,996 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-14 06:45:18,996 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30605.68 MB 2025-02-14 06:45:19,206 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 06:45:19,206 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 06:45:19,206 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 06:45:19,206 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:45:19,206 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29188.25 MB 2025-02-14 06:45:19,206 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31430.11 MB 2025-02-14 06:45:19,206 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 06:45:19,206 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33715.91 MB 2025-02-14 06:45:19,206 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39378.22 MB 2025-02-14 06:45:19,206 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 06:45:19,206 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36974.39 MB 2025-02-14 06:45:19,207 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 06:45:19,207 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 06:45:19,207 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 06:45:19,207 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:45:19,207 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27298.72 MB 2025-02-14 06:45:19,207 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31430.11 MB 2025-02-14 06:45:19,207 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 06:45:19,207 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30884.76 MB 2025-02-14 06:45:19,207 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39378.22 MB 2025-02-14 06:45:19,207 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8493.47 MB 2025-02-14 06:45:19,207 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36974.39 MB 2025-02-14 06:45:19,379 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 06:45:19,379 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 06:45:19,379 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 06:45:19,379 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:45:19,379 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32963.65 MB 2025-02-14 06:45:19,379 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33730.65 MB 2025-02-14 06:45:19,379 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 06:45:19,379 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39378.22 MB 2025-02-14 06:45:19,379 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39793.46 MB 2025-02-14 06:45:19,379 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 06:45:19,379 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34438.44 MB 2025-02-14 06:45:19,399 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 06:45:19,399 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 06:45:19,399 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:45:19,399 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:45:19,399 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34143.54 MB 2025-02-14 06:45:19,399 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34372.26 MB 2025-02-14 06:45:19,399 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.72 MB 2025-02-14 06:45:19,399 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39793.46 MB 2025-02-14 06:45:19,399 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39793.46 MB 2025-02-14 06:45:19,399 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:45:19,399 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34588.16 MB 2025-02-14 06:45:19,400 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 06:45:19,401 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 06:45:19,401 - resource_logging.py:150 - __exit__ - DEBUG - Time: 35.35 seconds 2025-02-14 06:45:19,401 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:45:19,401 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20334.05 MB 2025-02-14 06:45:19,401 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34572.89 MB 2025-02-14 06:45:19,401 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14238.84 MB 2025-02-14 06:45:19,401 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58864.96 MB 2025-02-14 06:45:19,401 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39793.46 MB 2025-02-14 06:45:19,401 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19071.50 MB 2025-02-14 06:45:19,401 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34588.16 MB 2025-02-14 06:45:19,673 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 06:45:19,673 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 06:45:19,673 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 06:45:19,673 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:45:19,673 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34572.89 MB 2025-02-14 06:45:19,673 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25332.47 MB 2025-02-14 06:45:19,673 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9240.42 MB 2025-02-14 06:45:19,673 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39793.46 MB 2025-02-14 06:45:19,673 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39793.46 MB 2025-02-14 06:45:19,673 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:45:19,673 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37079.02 MB 2025-02-14 06:45:19,691 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8144, cut from 8146 2025-02-14 06:45:19,692 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-14 06:45:19,698 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 06:45:19,698 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 06:45:19,698 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:45:19,698 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:45:19,698 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25332.47 MB 2025-02-14 06:45:19,698 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33753.25 MB 2025-02-14 06:45:19,699 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8420.78 MB 2025-02-14 06:45:19,699 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39793.46 MB 2025-02-14 06:45:19,699 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48165.29 MB 2025-02-14 06:45:19,699 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8371.83 MB 2025-02-14 06:45:19,699 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33753.25 MB 2025-02-14 06:45:19,864 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7936] 2025-02-14 06:45:19,866 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:45:19,866 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 06:45:19,867 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:45:19,867 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 06:45:19,872 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 06:45:19,873 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:45:19,873 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 06:45:19,873 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-14 06:46:33,481 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:46:33,482 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 06:46:33,489 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 06:46:33,497 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:46:33,497 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1340, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 06:46:33,499 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:46:33,499 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1340, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 06:46:54,190 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 06:46:54,190 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 06:46:54,190 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.68 seconds 2025-02-14 06:46:54,190 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:46:54,190 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22306.04 MB 2025-02-14 06:46:54,190 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27048.23 MB 2025-02-14 06:46:54,190 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4742.18 MB 2025-02-14 06:46:54,190 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56537.12 MB 2025-02-14 06:46:54,190 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38394.66 MB 2025-02-14 06:46:54,190 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18142.46 MB 2025-02-14 06:46:54,190 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35854.28 MB 2025-02-14 06:46:54,268 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 06:46:54,268 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 06:46:54,268 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 06:46:54,268 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:46:54,268 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27048.23 MB 2025-02-14 06:46:54,268 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22744.09 MB 2025-02-14 06:46:54,268 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4304.13 MB 2025-02-14 06:46:54,268 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38394.66 MB 2025-02-14 06:46:54,268 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47408.22 MB 2025-02-14 06:46:54,268 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9013.56 MB 2025-02-14 06:46:54,268 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40643.83 MB 2025-02-14 06:46:56,186 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 06:46:56,187 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 06:46:56,187 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 06:46:56,187 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:46:56,187 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22744.09 MB 2025-02-14 06:46:56,187 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23274.93 MB 2025-02-14 06:46:56,187 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 06:46:56,187 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47408.22 MB 2025-02-14 06:46:56,187 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29464.99 MB 2025-02-14 06:46:56,187 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17943.23 MB 2025-02-14 06:46:56,187 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27254.27 MB 2025-02-14 06:46:56,200 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 06:46:56,200 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 06:46:56,200 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:46:56,200 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:46:56,200 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23274.93 MB 2025-02-14 06:46:56,200 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25166.28 MB 2025-02-14 06:46:56,200 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1891.34 MB 2025-02-14 06:46:56,200 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29464.99 MB 2025-02-14 06:46:56,200 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29464.99 MB 2025-02-14 06:46:56,200 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:46:56,200 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26583.70 MB 2025-02-14 06:46:56,417 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 06:46:56,417 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 06:46:56,417 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 06:46:56,417 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:46:56,417 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25166.28 MB 2025-02-14 06:46:56,417 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27408.13 MB 2025-02-14 06:46:56,417 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 06:46:56,417 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29464.99 MB 2025-02-14 06:46:56,417 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35127.30 MB 2025-02-14 06:46:56,417 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 06:46:56,418 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32952.41 MB 2025-02-14 06:46:56,419 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 06:46:56,419 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 06:46:56,419 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 06:46:56,419 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:46:56,419 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23274.93 MB 2025-02-14 06:46:56,419 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27408.13 MB 2025-02-14 06:46:56,419 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4133.20 MB 2025-02-14 06:46:56,419 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29464.99 MB 2025-02-14 06:46:56,419 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35127.30 MB 2025-02-14 06:46:56,419 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 06:46:56,419 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32952.41 MB 2025-02-14 06:46:56,684 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 06:46:56,684 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 06:46:56,684 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 06:46:56,684 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:46:56,684 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28941.67 MB 2025-02-14 06:46:56,684 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29708.68 MB 2025-02-14 06:46:56,684 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 06:46:56,684 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35127.30 MB 2025-02-14 06:46:56,684 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35544.63 MB 2025-02-14 06:46:56,684 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 06:46:56,684 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30416.46 MB 2025-02-14 06:46:56,713 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 06:46:56,713 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 06:46:56,713 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:46:56,713 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:46:56,713 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30121.56 MB 2025-02-14 06:46:56,713 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30349.11 MB 2025-02-14 06:46:56,714 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.55 MB 2025-02-14 06:46:56,714 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35544.63 MB 2025-02-14 06:46:56,714 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35544.63 MB 2025-02-14 06:46:56,714 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:46:56,714 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30552.06 MB 2025-02-14 06:46:56,716 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 06:46:56,716 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 06:46:56,716 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.21 seconds 2025-02-14 06:46:56,716 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:46:56,716 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17637.37 MB 2025-02-14 06:46:56,716 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30549.87 MB 2025-02-14 06:46:56,716 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12912.49 MB 2025-02-14 06:46:56,716 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56537.12 MB 2025-02-14 06:46:56,716 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35544.63 MB 2025-02-14 06:46:56,716 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20992.49 MB 2025-02-14 06:46:56,716 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30552.06 MB 2025-02-14 06:46:57,004 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 06:46:57,004 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 06:46:57,004 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-14 06:46:57,004 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:46:57,004 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30549.87 MB 2025-02-14 06:46:57,004 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22636.81 MB 2025-02-14 06:46:57,004 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7913.05 MB 2025-02-14 06:46:57,004 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35544.63 MB 2025-02-14 06:46:57,004 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35544.63 MB 2025-02-14 06:46:57,004 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:46:57,004 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33057.54 MB 2025-02-14 06:46:57,023 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8149, cut from 8151 2025-02-14 06:46:57,023 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 06:46:57,030 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 06:46:57,030 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 06:46:57,030 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:46:57,030 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:46:57,030 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22636.81 MB 2025-02-14 06:46:57,030 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31062.99 MB 2025-02-14 06:46:57,030 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8426.18 MB 2025-02-14 06:46:57,030 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35544.63 MB 2025-02-14 06:46:57,030 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43920.65 MB 2025-02-14 06:46:57,030 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8376.03 MB 2025-02-14 06:46:57,030 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31062.99 MB 2025-02-14 06:46:57,197 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7941] 2025-02-14 06:46:57,199 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:46:57,199 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 06:46:57,200 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:46:57,200 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 06:46:57,204 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 06:46:57,205 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:46:57,205 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 06:46:57,205 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 06:47:29,981 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:47:29,981 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 06:47:29,986 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 06:47:29,991 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:47:29,991 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1592, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 06:47:29,992 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:47:29,992 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1592, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 06:47:54,694 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 06:47:54,694 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 06:47:54,694 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.69 seconds 2025-02-14 06:47:54,694 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:47:54,694 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24062.02 MB 2025-02-14 06:47:54,694 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29697.06 MB 2025-02-14 06:47:54,694 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5635.05 MB 2025-02-14 06:47:54,694 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52296.68 MB 2025-02-14 06:47:54,694 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39290.14 MB 2025-02-14 06:47:54,694 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13006.54 MB 2025-02-14 06:47:54,694 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38517.03 MB 2025-02-14 06:47:54,784 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 06:47:54,784 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 06:47:54,784 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 06:47:54,784 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:47:54,784 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29697.06 MB 2025-02-14 06:47:54,784 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24054.16 MB 2025-02-14 06:47:54,784 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5642.90 MB 2025-02-14 06:47:54,784 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39290.14 MB 2025-02-14 06:47:54,784 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51235.52 MB 2025-02-14 06:47:54,784 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 11945.38 MB 2025-02-14 06:47:54,784 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45784.74 MB 2025-02-14 06:47:56,716 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 06:47:56,716 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 06:47:56,716 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 06:47:56,716 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:47:56,716 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24054.16 MB 2025-02-14 06:47:56,716 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24585.00 MB 2025-02-14 06:47:56,716 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 06:47:56,716 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51235.52 MB 2025-02-14 06:47:56,716 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29469.18 MB 2025-02-14 06:47:56,716 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21766.34 MB 2025-02-14 06:47:56,716 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28565.38 MB 2025-02-14 06:47:56,730 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 06:47:56,730 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 06:47:56,730 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:47:56,730 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:47:56,730 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24585.00 MB 2025-02-14 06:47:56,730 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26474.54 MB 2025-02-14 06:47:56,730 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 06:47:56,730 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29469.18 MB 2025-02-14 06:47:56,730 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30412.90 MB 2025-02-14 06:47:56,730 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 06:47:56,730 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27891.97 MB 2025-02-14 06:47:56,964 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 06:47:56,964 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 06:47:56,964 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 06:47:56,964 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:47:56,964 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26474.54 MB 2025-02-14 06:47:56,964 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28716.39 MB 2025-02-14 06:47:56,964 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 06:47:56,964 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30412.90 MB 2025-02-14 06:47:56,964 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36547.07 MB 2025-02-14 06:47:56,964 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 06:47:56,964 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34260.68 MB 2025-02-14 06:47:56,965 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 06:47:56,965 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 06:47:56,965 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.25 seconds 2025-02-14 06:47:56,965 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:47:56,965 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24585.00 MB 2025-02-14 06:47:56,965 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28716.39 MB 2025-02-14 06:47:56,965 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 06:47:56,965 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29469.18 MB 2025-02-14 06:47:56,965 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36547.07 MB 2025-02-14 06:47:56,965 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7077.89 MB 2025-02-14 06:47:56,965 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34260.68 MB 2025-02-14 06:47:57,202 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 06:47:57,202 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 06:47:57,202 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 06:47:57,202 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:47:57,202 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30249.94 MB 2025-02-14 06:47:57,202 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31016.94 MB 2025-02-14 06:47:57,202 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 06:47:57,202 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36547.07 MB 2025-02-14 06:47:57,202 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36962.30 MB 2025-02-14 06:47:57,202 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 06:47:57,202 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31724.73 MB 2025-02-14 06:47:57,230 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 06:47:57,230 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 06:47:57,230 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:47:57,230 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:47:57,230 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31429.83 MB 2025-02-14 06:47:57,230 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31658.91 MB 2025-02-14 06:47:57,231 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.08 MB 2025-02-14 06:47:57,231 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36962.30 MB 2025-02-14 06:47:57,231 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36962.30 MB 2025-02-14 06:47:57,231 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:47:57,231 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31871.63 MB 2025-02-14 06:47:57,233 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 06:47:57,233 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 06:47:57,233 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.24 seconds 2025-02-14 06:47:57,233 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:47:57,233 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18515.36 MB 2025-02-14 06:47:57,233 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31859.91 MB 2025-02-14 06:47:57,233 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13344.55 MB 2025-02-14 06:47:57,233 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52296.68 MB 2025-02-14 06:47:57,233 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36962.30 MB 2025-02-14 06:47:57,233 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15334.38 MB 2025-02-14 06:47:57,233 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31871.63 MB 2025-02-14 06:47:57,522 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 06:47:57,522 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 06:47:57,522 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-14 06:47:57,522 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:47:57,522 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31859.91 MB 2025-02-14 06:47:57,522 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23518.61 MB 2025-02-14 06:47:57,522 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8341.30 MB 2025-02-14 06:47:57,522 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36962.30 MB 2025-02-14 06:47:57,522 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36962.30 MB 2025-02-14 06:47:57,522 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:47:57,522 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34370.66 MB 2025-02-14 06:47:57,540 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8159, cut from 8161 2025-02-14 06:47:57,540 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 06:47:57,547 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 06:47:57,547 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 06:47:57,547 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:47:57,547 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:47:57,547 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23518.61 MB 2025-02-14 06:47:57,547 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31954.20 MB 2025-02-14 06:47:57,547 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8435.59 MB 2025-02-14 06:47:57,547 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36962.30 MB 2025-02-14 06:47:57,547 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45350.91 MB 2025-02-14 06:47:57,547 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8388.61 MB 2025-02-14 06:47:57,547 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31954.20 MB 2025-02-14 06:47:57,723 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7951] 2025-02-14 06:47:57,725 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:47:57,725 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 06:47:57,726 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:47:57,726 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 06:47:57,731 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 06:47:57,732 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:47:57,732 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 06:47:57,732 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 06:50:03,824 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:50:03,824 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 06:50:03,832 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 06:50:03,839 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:50:03,839 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 852, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 06:50:03,841 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:50:03,841 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 852, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 06:50:17,048 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 06:50:17,048 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 06:50:17,048 - resource_logging.py:150 - __exit__ - DEBUG - Time: 13.20 seconds 2025-02-14 06:50:17,048 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:50:17,048 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18905.58 MB 2025-02-14 06:50:17,048 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21921.28 MB 2025-02-14 06:50:17,048 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3015.70 MB 2025-02-14 06:50:17,048 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53739.52 MB 2025-02-14 06:50:17,048 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28296.87 MB 2025-02-14 06:50:17,048 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25442.65 MB 2025-02-14 06:50:17,049 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30869.17 MB 2025-02-14 06:50:17,103 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 06:50:17,103 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 06:50:17,103 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-14 06:50:17,103 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:50:17,103 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21921.28 MB 2025-02-14 06:50:17,103 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20207.13 MB 2025-02-14 06:50:17,103 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1714.15 MB 2025-02-14 06:50:17,103 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28296.87 MB 2025-02-14 06:50:17,103 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37018.93 MB 2025-02-14 06:50:17,103 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8722.06 MB 2025-02-14 06:50:17,103 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32045.52 MB 2025-02-14 06:50:19,011 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 06:50:19,011 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 06:50:19,011 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 06:50:19,011 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:50:19,011 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20207.13 MB 2025-02-14 06:50:19,011 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20737.97 MB 2025-02-14 06:50:19,011 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 06:50:19,011 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37018.93 MB 2025-02-14 06:50:19,011 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26696.74 MB 2025-02-14 06:50:19,011 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10322.18 MB 2025-02-14 06:50:19,011 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24717.31 MB 2025-02-14 06:50:19,025 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 06:50:19,025 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 06:50:19,025 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:50:19,025 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:50:19,025 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20737.97 MB 2025-02-14 06:50:19,025 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22627.51 MB 2025-02-14 06:50:19,025 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 06:50:19,025 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26696.74 MB 2025-02-14 06:50:19,025 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26696.74 MB 2025-02-14 06:50:19,025 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:50:19,025 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24044.94 MB 2025-02-14 06:50:19,240 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 06:50:19,240 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 06:50:19,240 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 06:50:19,240 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:50:19,240 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22627.51 MB 2025-02-14 06:50:19,240 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24869.36 MB 2025-02-14 06:50:19,241 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 06:50:19,241 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26696.74 MB 2025-02-14 06:50:19,241 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32830.91 MB 2025-02-14 06:50:19,241 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 06:50:19,241 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30413.65 MB 2025-02-14 06:50:19,241 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 06:50:19,241 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 06:50:19,241 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 06:50:19,241 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:50:19,241 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20737.97 MB 2025-02-14 06:50:19,241 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24869.36 MB 2025-02-14 06:50:19,241 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 06:50:19,241 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26696.74 MB 2025-02-14 06:50:19,241 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32830.91 MB 2025-02-14 06:50:19,241 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 06:50:19,241 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30413.65 MB 2025-02-14 06:50:19,421 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 06:50:19,421 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 06:50:19,421 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 06:50:19,421 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:50:19,421 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26402.91 MB 2025-02-14 06:50:19,421 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27169.91 MB 2025-02-14 06:50:19,421 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 06:50:19,421 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32830.91 MB 2025-02-14 06:50:19,421 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33246.15 MB 2025-02-14 06:50:19,421 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 06:50:19,421 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27877.70 MB 2025-02-14 06:50:19,442 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 06:50:19,442 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 06:50:19,442 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:50:19,442 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:50:19,442 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27582.80 MB 2025-02-14 06:50:19,442 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27810.34 MB 2025-02-14 06:50:19,442 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.55 MB 2025-02-14 06:50:19,443 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33246.15 MB 2025-02-14 06:50:19,443 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33246.15 MB 2025-02-14 06:50:19,443 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:50:19,443 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28031.54 MB 2025-02-14 06:50:19,444 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 06:50:19,444 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 06:50:19,444 - resource_logging.py:150 - __exit__ - DEBUG - Time: 15.60 seconds 2025-02-14 06:50:19,444 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:50:19,444 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15937.14 MB 2025-02-14 06:50:19,444 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28011.10 MB 2025-02-14 06:50:19,444 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12073.96 MB 2025-02-14 06:50:19,444 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53739.52 MB 2025-02-14 06:50:19,444 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33246.15 MB 2025-02-14 06:50:19,444 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20493.37 MB 2025-02-14 06:50:19,444 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28031.54 MB 2025-02-14 06:50:19,712 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 06:50:19,712 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 06:50:19,712 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 06:50:19,712 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:50:19,712 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28011.10 MB 2025-02-14 06:50:19,712 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20936.58 MB 2025-02-14 06:50:19,712 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7074.52 MB 2025-02-14 06:50:19,712 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33246.15 MB 2025-02-14 06:50:19,712 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33246.15 MB 2025-02-14 06:50:19,712 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:50:19,712 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30518.77 MB 2025-02-14 06:50:19,730 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8149, cut from 8151 2025-02-14 06:50:19,730 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2 ('] 2025-02-14 06:50:19,736 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 06:50:19,737 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 06:50:19,737 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:50:19,737 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:50:19,737 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20936.58 MB 2025-02-14 06:50:19,737 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29362.76 MB 2025-02-14 06:50:19,737 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8426.18 MB 2025-02-14 06:50:19,737 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33246.15 MB 2025-02-14 06:50:19,737 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41622.18 MB 2025-02-14 06:50:19,737 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8376.03 MB 2025-02-14 06:50:19,737 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29362.76 MB 2025-02-14 06:50:19,906 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7941] 2025-02-14 06:50:19,908 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:50:19,908 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 06:50:19,909 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:50:19,909 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 06:50:19,914 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 06:50:19,915 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:50:19,915 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 06:50:19,915 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2 ('] 2025-02-14 06:52:38,405 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:52:38,405 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 06:52:38,410 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 06:52:38,414 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:52:38,414 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2364, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 06:52:38,415 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:52:38,415 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2364, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 06:53:14,680 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 06:53:14,680 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 06:53:14,680 - resource_logging.py:150 - __exit__ - DEBUG - Time: 36.26 seconds 2025-02-14 06:53:14,680 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:53:14,680 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29441.44 MB 2025-02-14 06:53:14,680 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37807.50 MB 2025-02-14 06:53:14,680 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8366.06 MB 2025-02-14 06:53:14,680 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49998.20 MB 2025-02-14 06:53:14,680 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42024.83 MB 2025-02-14 06:53:14,680 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7973.37 MB 2025-02-14 06:53:14,680 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46614.36 MB 2025-02-14 06:53:14,829 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 06:53:14,829 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 06:53:14,829 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 06:53:14,829 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:53:14,829 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37807.50 MB 2025-02-14 06:53:14,829 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28067.55 MB 2025-02-14 06:53:14,829 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9739.95 MB 2025-02-14 06:53:14,829 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42024.83 MB 2025-02-14 06:53:14,829 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 72540.49 MB 2025-02-14 06:53:14,829 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 30515.66 MB 2025-02-14 06:53:14,829 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 61471.29 MB 2025-02-14 06:53:16,804 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 06:53:16,804 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 06:53:16,804 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.97 seconds 2025-02-14 06:53:16,804 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:53:16,804 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28067.55 MB 2025-02-14 06:53:16,804 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28598.39 MB 2025-02-14 06:53:16,805 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 06:53:16,805 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 72540.49 MB 2025-02-14 06:53:16,805 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30886.85 MB 2025-02-14 06:53:16,805 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -41653.63 MB 2025-02-14 06:53:16,805 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32578.76 MB 2025-02-14 06:53:16,819 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 06:53:16,819 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 06:53:16,819 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:53:16,819 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:53:16,819 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28598.39 MB 2025-02-14 06:53:16,819 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30487.92 MB 2025-02-14 06:53:16,819 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 06:53:16,819 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30886.85 MB 2025-02-14 06:53:16,819 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34189.87 MB 2025-02-14 06:53:16,819 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 06:53:16,819 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31905.35 MB 2025-02-14 06:53:17,027 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 06:53:17,027 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 06:53:17,027 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 06:53:17,027 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:53:17,027 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30487.92 MB 2025-02-14 06:53:17,027 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32729.78 MB 2025-02-14 06:53:17,027 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 06:53:17,027 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34189.87 MB 2025-02-14 06:53:17,027 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40795.90 MB 2025-02-14 06:53:17,027 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 06:53:17,027 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38274.06 MB 2025-02-14 06:53:17,028 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 06:53:17,028 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 06:53:17,028 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 06:53:17,028 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:53:17,028 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28598.39 MB 2025-02-14 06:53:17,028 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32729.78 MB 2025-02-14 06:53:17,028 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 06:53:17,028 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30886.85 MB 2025-02-14 06:53:17,028 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40795.90 MB 2025-02-14 06:53:17,028 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9909.04 MB 2025-02-14 06:53:17,028 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38274.06 MB 2025-02-14 06:53:17,190 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 06:53:17,190 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 06:53:17,190 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 06:53:17,190 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:53:17,190 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34263.32 MB 2025-02-14 06:53:17,190 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35030.32 MB 2025-02-14 06:53:17,190 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 06:53:17,190 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40795.90 MB 2025-02-14 06:53:17,190 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41211.13 MB 2025-02-14 06:53:17,190 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 06:53:17,190 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35738.11 MB 2025-02-14 06:53:17,208 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 06:53:17,208 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 06:53:17,208 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:53:17,208 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:53:17,208 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35443.21 MB 2025-02-14 06:53:17,208 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35671.46 MB 2025-02-14 06:53:17,208 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.25 MB 2025-02-14 06:53:17,208 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41211.13 MB 2025-02-14 06:53:17,208 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41211.13 MB 2025-02-14 06:53:17,208 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:53:17,208 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35890.05 MB 2025-02-14 06:53:17,210 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 06:53:17,210 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 06:53:17,210 - resource_logging.py:150 - __exit__ - DEBUG - Time: 38.79 seconds 2025-02-14 06:53:17,210 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:53:17,210 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21205.07 MB 2025-02-14 06:53:17,210 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35871.63 MB 2025-02-14 06:53:17,210 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14666.55 MB 2025-02-14 06:53:17,210 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49998.20 MB 2025-02-14 06:53:17,210 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41211.13 MB 2025-02-14 06:53:17,210 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8787.07 MB 2025-02-14 06:53:17,210 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35890.05 MB 2025-02-14 06:53:17,477 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 06:53:17,478 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 06:53:17,478 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 06:53:17,478 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:53:17,478 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35871.63 MB 2025-02-14 06:53:17,478 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26195.37 MB 2025-02-14 06:53:17,478 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9676.26 MB 2025-02-14 06:53:17,478 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41211.13 MB 2025-02-14 06:53:17,478 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41211.13 MB 2025-02-14 06:53:17,478 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:53:17,478 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38371.93 MB 2025-02-14 06:53:17,495 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8125, cut from 8127 2025-02-14 06:53:17,496 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 1 ('] 2025-02-14 06:53:17,502 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 06:53:17,502 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 06:53:17,502 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:53:17,502 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:53:17,502 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26195.37 MB 2025-02-14 06:53:17,502 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34596.31 MB 2025-02-14 06:53:17,502 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8400.94 MB 2025-02-14 06:53:17,502 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41211.13 MB 2025-02-14 06:53:17,502 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49564.09 MB 2025-02-14 06:53:17,502 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8352.96 MB 2025-02-14 06:53:17,502 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34596.31 MB 2025-02-14 06:53:17,652 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7917] 2025-02-14 06:53:17,653 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:53:17,653 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 06:53:17,654 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:53:17,654 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 06:53:17,658 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 06:53:17,659 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:53:17,660 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 06:53:17,660 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 1 ('] 2025-02-14 06:54:17,218 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:54:17,219 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 06:54:17,223 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 06:54:17,227 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:54:17,227 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 3297, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 06:54:17,228 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:54:17,228 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 3297, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 06:55:08,575 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 06:55:08,575 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 06:55:08,575 - resource_logging.py:150 - __exit__ - DEBUG - Time: 51.33 seconds 2025-02-14 06:55:08,575 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:55:08,575 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35945.10 MB 2025-02-14 06:55:08,575 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47613.66 MB 2025-02-14 06:55:08,575 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11668.55 MB 2025-02-14 06:55:08,575 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 85068.87 MB 2025-02-14 06:55:08,575 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51556.38 MB 2025-02-14 06:55:08,575 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33512.49 MB 2025-02-14 06:55:08,575 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 59281.56 MB 2025-02-14 06:55:08,807 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 06:55:08,807 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 06:55:08,807 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 06:55:08,807 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:55:08,807 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47613.66 MB 2025-02-14 06:55:08,807 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32919.97 MB 2025-02-14 06:55:08,807 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -14693.69 MB 2025-02-14 06:55:08,807 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51556.38 MB 2025-02-14 06:55:08,807 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 95793.71 MB 2025-02-14 06:55:08,807 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 44237.32 MB 2025-02-14 06:55:08,807 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 81199.39 MB 2025-02-14 06:55:10,810 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 06:55:10,810 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 06:55:10,810 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.00 seconds 2025-02-14 06:55:10,810 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:55:10,810 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32919.97 MB 2025-02-14 06:55:10,810 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33450.81 MB 2025-02-14 06:55:10,810 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 06:55:10,810 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 95793.71 MB 2025-02-14 06:55:10,810 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35469.13 MB 2025-02-14 06:55:10,810 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -60324.58 MB 2025-02-14 06:55:10,810 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37431.18 MB 2025-02-14 06:55:10,824 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 06:55:10,825 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 06:55:10,825 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:55:10,825 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:55:10,825 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33450.81 MB 2025-02-14 06:55:10,825 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35340.35 MB 2025-02-14 06:55:10,825 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 06:55:10,825 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35469.13 MB 2025-02-14 06:55:10,825 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38772.15 MB 2025-02-14 06:55:10,825 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 06:55:10,825 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36757.78 MB 2025-02-14 06:55:11,071 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 06:55:11,071 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 06:55:11,071 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.24 seconds 2025-02-14 06:55:11,071 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:55:11,071 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35340.35 MB 2025-02-14 06:55:11,071 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37582.20 MB 2025-02-14 06:55:11,071 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 06:55:11,071 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38772.15 MB 2025-02-14 06:55:11,071 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45378.17 MB 2025-02-14 06:55:11,071 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 06:55:11,071 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43126.48 MB 2025-02-14 06:55:11,072 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 06:55:11,072 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 06:55:11,072 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 06:55:11,072 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:55:11,072 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33450.81 MB 2025-02-14 06:55:11,072 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37582.20 MB 2025-02-14 06:55:11,072 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 06:55:11,073 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35469.13 MB 2025-02-14 06:55:11,073 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45378.17 MB 2025-02-14 06:55:11,073 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9909.04 MB 2025-02-14 06:55:11,073 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43126.48 MB 2025-02-14 06:55:11,342 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 06:55:11,342 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 06:55:11,342 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 06:55:11,342 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:55:11,342 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39115.74 MB 2025-02-14 06:55:11,342 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39882.75 MB 2025-02-14 06:55:11,342 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 06:55:11,342 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45378.17 MB 2025-02-14 06:55:11,342 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45791.31 MB 2025-02-14 06:55:11,342 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 06:55:11,342 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40590.54 MB 2025-02-14 06:55:11,372 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 06:55:11,372 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 06:55:11,372 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 06:55:11,372 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:55:11,372 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40295.64 MB 2025-02-14 06:55:11,372 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40524.62 MB 2025-02-14 06:55:11,372 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.99 MB 2025-02-14 06:55:11,372 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45791.31 MB 2025-02-14 06:55:11,372 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45791.31 MB 2025-02-14 06:55:11,373 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:55:11,373 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40739.49 MB 2025-02-14 06:55:11,375 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 06:55:11,375 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 06:55:11,375 - resource_logging.py:150 - __exit__ - DEBUG - Time: 54.14 seconds 2025-02-14 06:55:11,375 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:55:11,375 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24456.90 MB 2025-02-14 06:55:11,375 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40725.64 MB 2025-02-14 06:55:11,375 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 16268.74 MB 2025-02-14 06:55:11,375 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 73580.68 MB 2025-02-14 06:55:11,375 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45791.31 MB 2025-02-14 06:55:11,375 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27789.36 MB 2025-02-14 06:55:11,375 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40739.49 MB 2025-02-14 06:55:11,663 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 06:55:11,663 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 06:55:11,663 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-14 06:55:11,663 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:55:11,663 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40725.64 MB 2025-02-14 06:55:11,663 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29460.53 MB 2025-02-14 06:55:11,663 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -11265.11 MB 2025-02-14 06:55:11,663 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45791.31 MB 2025-02-14 06:55:11,663 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45791.31 MB 2025-02-14 06:55:11,663 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:55:11,663 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43236.70 MB 2025-02-14 06:55:11,681 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8160, cut from 8162 2025-02-14 06:55:11,681 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 06:55:11,687 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 06:55:11,687 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 06:55:11,687 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:55:11,687 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:55:11,687 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29460.53 MB 2025-02-14 06:55:11,687 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37898.00 MB 2025-02-14 06:55:11,687 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8437.47 MB 2025-02-14 06:55:11,687 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45791.31 MB 2025-02-14 06:55:11,687 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49985.62 MB 2025-02-14 06:55:11,687 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4194.30 MB 2025-02-14 06:55:11,687 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37898.00 MB 2025-02-14 06:55:11,850 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7952] 2025-02-14 06:55:11,852 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:55:11,852 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 06:55:11,853 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:55:11,853 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 06:55:11,857 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 06:55:11,858 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:55:11,858 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 06:55:11,859 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 06:55:26,825 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:55:26,825 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 06:55:26,833 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 06:55:26,839 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:55:26,839 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1341, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 06:55:26,841 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:55:26,841 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1341, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 06:55:47,903 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 06:55:47,903 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 06:55:47,903 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.05 seconds 2025-02-14 06:55:47,903 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:55:47,903 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22313.01 MB 2025-02-14 06:55:47,903 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27058.86 MB 2025-02-14 06:55:47,903 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4745.85 MB 2025-02-14 06:55:47,903 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58374.23 MB 2025-02-14 06:55:47,903 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39894.12 MB 2025-02-14 06:55:47,903 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18480.10 MB 2025-02-14 06:55:47,903 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35861.24 MB 2025-02-14 06:55:47,979 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 06:55:47,979 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 06:55:47,979 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 06:55:47,979 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:55:47,979 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27058.86 MB 2025-02-14 06:55:47,979 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22749.29 MB 2025-02-14 06:55:47,979 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4309.57 MB 2025-02-14 06:55:47,979 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39894.12 MB 2025-02-14 06:55:47,979 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46684.70 MB 2025-02-14 06:55:47,979 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6790.58 MB 2025-02-14 06:55:47,979 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40752.51 MB 2025-02-14 06:55:49,918 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 06:55:49,918 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 06:55:49,918 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-14 06:55:49,918 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:55:49,918 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22749.29 MB 2025-02-14 06:55:49,918 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23280.13 MB 2025-02-14 06:55:49,918 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 06:55:49,918 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46684.70 MB 2025-02-14 06:55:49,918 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35148.27 MB 2025-02-14 06:55:49,918 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11536.43 MB 2025-02-14 06:55:49,918 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27259.47 MB 2025-02-14 06:55:49,932 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 06:55:49,932 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 06:55:49,932 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:55:49,932 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:55:49,932 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23280.13 MB 2025-02-14 06:55:49,932 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25169.67 MB 2025-02-14 06:55:49,932 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 06:55:49,932 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35148.27 MB 2025-02-14 06:55:49,932 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35148.27 MB 2025-02-14 06:55:49,932 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:55:49,932 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26587.10 MB 2025-02-14 06:55:50,143 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 06:55:50,143 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 06:55:50,143 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 06:55:50,144 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:55:50,144 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25169.67 MB 2025-02-14 06:55:50,144 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27411.52 MB 2025-02-14 06:55:50,144 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 06:55:50,144 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35148.27 MB 2025-02-14 06:55:50,144 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36563.85 MB 2025-02-14 06:55:50,144 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1415.58 MB 2025-02-14 06:55:50,144 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32955.80 MB 2025-02-14 06:55:50,144 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 06:55:50,144 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 06:55:50,144 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 06:55:50,144 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:55:50,144 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23280.13 MB 2025-02-14 06:55:50,144 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27411.52 MB 2025-02-14 06:55:50,144 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 06:55:50,144 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35148.27 MB 2025-02-14 06:55:50,144 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36563.85 MB 2025-02-14 06:55:50,144 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1415.58 MB 2025-02-14 06:55:50,144 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32955.80 MB 2025-02-14 06:55:50,311 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 06:55:50,311 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 06:55:50,311 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 06:55:50,311 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:55:50,311 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28945.07 MB 2025-02-14 06:55:50,311 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29712.07 MB 2025-02-14 06:55:50,311 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 06:55:50,311 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36563.85 MB 2025-02-14 06:55:50,311 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36976.98 MB 2025-02-14 06:55:50,311 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 06:55:50,311 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30419.86 MB 2025-02-14 06:55:50,330 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 06:55:50,330 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 06:55:50,330 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:55:50,330 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:55:50,330 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30124.96 MB 2025-02-14 06:55:50,330 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30353.04 MB 2025-02-14 06:55:50,330 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.08 MB 2025-02-14 06:55:50,330 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36976.98 MB 2025-02-14 06:55:50,330 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36976.98 MB 2025-02-14 06:55:50,330 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:55:50,330 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30565.32 MB 2025-02-14 06:55:50,331 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 06:55:50,331 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 06:55:50,331 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.49 seconds 2025-02-14 06:55:50,331 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:55:50,331 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17640.86 MB 2025-02-14 06:55:50,331 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30554.04 MB 2025-02-14 06:55:50,331 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12913.18 MB 2025-02-14 06:55:50,331 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58374.23 MB 2025-02-14 06:55:50,331 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36976.98 MB 2025-02-14 06:55:50,331 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21397.24 MB 2025-02-14 06:55:50,331 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30565.32 MB 2025-02-14 06:55:50,605 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 06:55:50,605 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 06:55:50,605 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 06:55:50,605 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:55:50,605 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30554.04 MB 2025-02-14 06:55:50,605 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22644.10 MB 2025-02-14 06:55:50,605 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7909.93 MB 2025-02-14 06:55:50,605 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36976.98 MB 2025-02-14 06:55:50,605 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36976.98 MB 2025-02-14 06:55:50,605 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:55:50,605 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33064.78 MB 2025-02-14 06:55:50,626 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8159, cut from 8161 2025-02-14 06:55:50,626 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-14 06:55:50,638 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 06:55:50,638 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 06:55:50,638 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 06:55:50,638 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:55:50,638 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22644.10 MB 2025-02-14 06:55:50,638 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31079.70 MB 2025-02-14 06:55:50,638 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8435.59 MB 2025-02-14 06:55:50,638 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36976.98 MB 2025-02-14 06:55:50,638 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45365.59 MB 2025-02-14 06:55:50,638 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8388.61 MB 2025-02-14 06:55:50,638 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31079.70 MB 2025-02-14 06:55:50,805 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7951] 2025-02-14 06:55:50,806 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:55:50,806 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 06:55:50,807 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:55:50,807 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 06:55:50,812 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 06:55:50,813 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:55:50,813 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 06:55:50,813 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-14 06:55:59,770 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:55:59,770 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 06:55:59,775 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 06:55:59,779 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:55:59,779 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 233, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 06:55:59,780 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:55:59,780 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 233, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 06:56:03,441 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 06:56:03,441 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 06:56:03,441 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.66 seconds 2025-02-14 06:56:03,441 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:56:03,441 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14592.29 MB 2025-02-14 06:56:03,441 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15416.86 MB 2025-02-14 06:56:03,441 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 824.57 MB 2025-02-14 06:56:03,441 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53754.20 MB 2025-02-14 06:56:03,441 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18433.97 MB 2025-02-14 06:56:03,442 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35320.23 MB 2025-02-14 06:56:03,442 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24290.96 MB 2025-02-14 06:56:03,458 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 06:56:03,458 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 06:56:03,458 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:56:03,458 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:56:03,458 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15416.86 MB 2025-02-14 06:56:03,458 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15739.83 MB 2025-02-14 06:56:03,458 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 322.97 MB 2025-02-14 06:56:03,458 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18433.97 MB 2025-02-14 06:56:03,458 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20006.83 MB 2025-02-14 06:56:03,458 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1572.86 MB 2025-02-14 06:56:03,458 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18568.51 MB 2025-02-14 06:56:04,535 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 06:56:04,535 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 06:56:04,536 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.08 seconds 2025-02-14 06:56:04,536 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:56:04,536 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15739.83 MB 2025-02-14 06:56:04,536 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16034.45 MB 2025-02-14 06:56:04,536 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 294.62 MB 2025-02-14 06:56:04,536 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20006.83 MB 2025-02-14 06:56:04,536 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18100.52 MB 2025-02-14 06:56:04,536 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1906.31 MB 2025-02-14 06:56:04,536 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19996.24 MB 2025-02-14 06:56:04,545 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 06:56:04,545 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 06:56:04,545 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:56:04,545 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:56:04,545 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16034.45 MB 2025-02-14 06:56:04,545 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17083.94 MB 2025-02-14 06:56:04,545 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1049.49 MB 2025-02-14 06:56:04,545 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18100.52 MB 2025-02-14 06:56:04,545 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19149.09 MB 2025-02-14 06:56:04,545 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1048.58 MB 2025-02-14 06:56:04,545 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17870.61 MB 2025-02-14 06:56:04,663 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 06:56:04,663 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 06:56:04,663 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 06:56:04,663 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:56:04,663 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17083.94 MB 2025-02-14 06:56:04,663 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18328.98 MB 2025-02-14 06:56:04,663 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1245.05 MB 2025-02-14 06:56:04,663 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19149.09 MB 2025-02-14 06:56:04,663 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22556.97 MB 2025-02-14 06:56:04,664 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3407.87 MB 2025-02-14 06:56:04,664 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21407.86 MB 2025-02-14 06:56:04,664 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 06:56:04,664 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 06:56:04,664 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 06:56:04,664 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:56:04,664 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16034.45 MB 2025-02-14 06:56:04,664 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18328.98 MB 2025-02-14 06:56:04,664 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2294.53 MB 2025-02-14 06:56:04,664 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18100.52 MB 2025-02-14 06:56:04,664 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22556.97 MB 2025-02-14 06:56:04,664 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4456.45 MB 2025-02-14 06:56:04,664 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21407.86 MB 2025-02-14 06:56:04,758 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 06:56:04,758 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 06:56:04,758 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 06:56:04,758 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:56:04,758 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19180.10 MB 2025-02-14 06:56:04,758 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19605.78 MB 2025-02-14 06:56:04,758 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 425.69 MB 2025-02-14 06:56:04,758 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22556.97 MB 2025-02-14 06:56:04,758 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22783.46 MB 2025-02-14 06:56:04,758 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 226.49 MB 2025-02-14 06:56:04,758 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19999.43 MB 2025-02-14 06:56:04,769 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 06:56:04,770 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 06:56:04,770 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:56:04,770 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:56:04,770 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19834.94 MB 2025-02-14 06:56:04,770 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20055.11 MB 2025-02-14 06:56:04,770 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 220.17 MB 2025-02-14 06:56:04,770 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22783.46 MB 2025-02-14 06:56:04,770 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22783.46 MB 2025-02-14 06:56:04,770 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:56:04,770 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20134.29 MB 2025-02-14 06:56:04,771 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 06:56:04,771 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 06:56:04,771 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.99 seconds 2025-02-14 06:56:04,771 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:56:04,771 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13780.50 MB 2025-02-14 06:56:04,771 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20255.76 MB 2025-02-14 06:56:04,771 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6475.27 MB 2025-02-14 06:56:04,771 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53754.20 MB 2025-02-14 06:56:04,771 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22783.46 MB 2025-02-14 06:56:04,771 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30970.74 MB 2025-02-14 06:56:04,771 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20255.76 MB 2025-02-14 06:56:05,042 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 06:56:05,042 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 06:56:05,042 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 06:56:05,042 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:56:05,042 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14930.61 MB 2025-02-14 06:56:05,042 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17938.38 MB 2025-02-14 06:56:05,042 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3007.77 MB 2025-02-14 06:56:05,042 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22783.46 MB 2025-02-14 06:56:05,042 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22783.46 MB 2025-02-14 06:56:05,042 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:56:05,042 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18239.12 MB 2025-02-14 06:56:05,060 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8145, cut from 8147 2025-02-14 06:56:05,060 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 06:56:05,066 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 06:56:05,066 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 06:56:05,066 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:56:05,066 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:56:05,066 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17938.38 MB 2025-02-14 06:56:05,066 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26360.34 MB 2025-02-14 06:56:05,066 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8421.96 MB 2025-02-14 06:56:05,066 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22783.46 MB 2025-02-14 06:56:05,066 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33248.25 MB 2025-02-14 06:56:05,066 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10464.79 MB 2025-02-14 06:56:05,066 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26360.34 MB 2025-02-14 06:56:05,232 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7937] 2025-02-14 06:56:05,233 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:56:05,233 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 06:56:05,234 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:56:05,234 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 06:56:05,239 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 06:56:05,240 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:56:05,240 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 06:56:05,240 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 06:57:02,801 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:57:02,802 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 06:57:02,807 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 06:57:02,810 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:57:02,810 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 112, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 06:57:02,811 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:57:02,811 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 112, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 06:57:04,540 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 06:57:04,540 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 06:57:04,540 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.72 seconds 2025-02-14 06:57:04,540 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:57:04,540 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13749.14 MB 2025-02-14 06:57:04,540 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14145.50 MB 2025-02-14 06:57:04,540 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 396.36 MB 2025-02-14 06:57:04,540 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41620.08 MB 2025-02-14 06:57:04,540 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17574.13 MB 2025-02-14 06:57:04,540 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24045.94 MB 2025-02-14 06:57:04,540 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22994.02 MB 2025-02-14 06:57:04,543 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 06:57:04,543 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 06:57:04,543 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 06:57:04,543 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:57:04,543 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14145.50 MB 2025-02-14 06:57:04,543 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14337.54 MB 2025-02-14 06:57:04,543 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 192.04 MB 2025-02-14 06:57:04,543 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17574.13 MB 2025-02-14 06:57:04,543 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17574.13 MB 2025-02-14 06:57:04,543 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:57:04,543 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14932.15 MB 2025-02-14 06:57:05,085 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 06:57:05,085 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 06:57:05,085 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.54 seconds 2025-02-14 06:57:05,085 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:57:05,085 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14337.54 MB 2025-02-14 06:57:05,085 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14486.17 MB 2025-02-14 06:57:05,085 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 148.64 MB 2025-02-14 06:57:05,085 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17574.13 MB 2025-02-14 06:57:05,085 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17574.13 MB 2025-02-14 06:57:05,085 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:57:05,085 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18424.08 MB 2025-02-14 06:57:05,091 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 06:57:05,091 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 06:57:05,091 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 06:57:05,091 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:57:05,091 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14486.11 MB 2025-02-14 06:57:05,091 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15015.05 MB 2025-02-14 06:57:05,091 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 528.94 MB 2025-02-14 06:57:05,091 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17574.13 MB 2025-02-14 06:57:05,091 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17574.13 MB 2025-02-14 06:57:05,091 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:57:05,091 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15411.94 MB 2025-02-14 06:57:05,203 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 06:57:05,203 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 06:57:05,203 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 06:57:05,203 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:57:05,203 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15015.05 MB 2025-02-14 06:57:05,203 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15658.01 MB 2025-02-14 06:57:05,203 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 642.96 MB 2025-02-14 06:57:05,203 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17574.13 MB 2025-02-14 06:57:05,203 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18102.62 MB 2025-02-14 06:57:05,203 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 528.48 MB 2025-02-14 06:57:05,203 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17195.17 MB 2025-02-14 06:57:05,204 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 06:57:05,204 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 06:57:05,204 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 06:57:05,204 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:57:05,204 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14486.11 MB 2025-02-14 06:57:05,204 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15658.01 MB 2025-02-14 06:57:05,204 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1171.90 MB 2025-02-14 06:57:05,204 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17574.13 MB 2025-02-14 06:57:05,204 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18102.62 MB 2025-02-14 06:57:05,204 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 528.48 MB 2025-02-14 06:57:05,204 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17195.17 MB 2025-02-14 06:57:05,261 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 06:57:05,261 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 06:57:05,261 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-14 06:57:05,261 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:57:05,261 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16278.37 MB 2025-02-14 06:57:05,261 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16548.18 MB 2025-02-14 06:57:05,261 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 269.81 MB 2025-02-14 06:57:05,261 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18102.62 MB 2025-02-14 06:57:05,261 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18274.58 MB 2025-02-14 06:57:05,261 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 171.97 MB 2025-02-14 06:57:05,261 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16746.36 MB 2025-02-14 06:57:05,268 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 06:57:05,268 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 06:57:05,268 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 06:57:05,268 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:57:05,268 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16718.85 MB 2025-02-14 06:57:05,268 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16948.01 MB 2025-02-14 06:57:05,268 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.16 MB 2025-02-14 06:57:05,268 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18274.58 MB 2025-02-14 06:57:05,268 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18274.58 MB 2025-02-14 06:57:05,268 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:57:05,268 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16948.01 MB 2025-02-14 06:57:05,269 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 06:57:05,269 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 06:57:05,269 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.46 seconds 2025-02-14 06:57:05,269 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:57:05,269 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13358.92 MB 2025-02-14 06:57:05,269 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17148.42 MB 2025-02-14 06:57:05,269 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3789.50 MB 2025-02-14 06:57:05,269 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41620.08 MB 2025-02-14 06:57:05,269 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18274.58 MB 2025-02-14 06:57:05,269 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23345.50 MB 2025-02-14 06:57:05,269 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17148.42 MB 2025-02-14 06:57:05,538 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 06:57:05,538 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 06:57:05,538 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 06:57:05,538 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:57:05,538 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17148.42 MB 2025-02-14 06:57:05,538 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20152.50 MB 2025-02-14 06:57:05,538 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3004.08 MB 2025-02-14 06:57:05,538 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18274.58 MB 2025-02-14 06:57:05,539 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21764.24 MB 2025-02-14 06:57:05,539 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3489.66 MB 2025-02-14 06:57:05,539 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20453.58 MB 2025-02-14 06:57:05,557 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8135, cut from 8137 2025-02-14 06:57:05,557 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 06:57:05,563 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 06:57:05,563 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 06:57:05,563 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:57:05,563 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:57:05,563 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20152.50 MB 2025-02-14 06:57:05,563 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28563.32 MB 2025-02-14 06:57:05,563 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8410.82 MB 2025-02-14 06:57:05,563 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21764.24 MB 2025-02-14 06:57:05,563 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32218.55 MB 2025-02-14 06:57:05,563 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10454.30 MB 2025-02-14 06:57:05,563 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28563.32 MB 2025-02-14 06:57:05,725 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7927] 2025-02-14 06:57:05,726 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:57:05,726 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 06:57:05,727 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:57:05,727 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 06:57:05,732 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 06:57:05,733 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:57:05,733 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 06:57:05,733 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 06:58:22,152 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:58:22,152 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 06:58:22,157 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 06:58:22,161 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:58:22,161 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1104, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 06:58:22,162 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:58:22,162 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1104, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 06:58:39,058 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 06:58:39,058 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 06:58:39,058 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.89 seconds 2025-02-14 06:58:39,058 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:58:39,058 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20661.56 MB 2025-02-14 06:58:39,058 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24568.55 MB 2025-02-14 06:58:39,058 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3906.99 MB 2025-02-14 06:58:39,058 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40584.09 MB 2025-02-14 06:58:39,058 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31274.83 MB 2025-02-14 06:58:39,058 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9309.26 MB 2025-02-14 06:58:39,058 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33530.31 MB 2025-02-14 06:58:39,121 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 06:58:39,121 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 06:58:39,121 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 06:58:39,121 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:58:39,121 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24568.55 MB 2025-02-14 06:58:39,121 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21517.20 MB 2025-02-14 06:58:39,121 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3051.35 MB 2025-02-14 06:58:39,121 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31274.83 MB 2025-02-14 06:58:39,121 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40451.96 MB 2025-02-14 06:58:39,121 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9177.14 MB 2025-02-14 06:58:39,121 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35973.01 MB 2025-02-14 06:58:41,023 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 06:58:41,023 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 06:58:41,023 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.90 seconds 2025-02-14 06:58:41,023 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:58:41,023 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21517.20 MB 2025-02-14 06:58:41,023 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22048.04 MB 2025-02-14 06:58:41,023 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 06:58:41,023 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40451.96 MB 2025-02-14 06:58:41,023 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28783.41 MB 2025-02-14 06:58:41,023 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11668.55 MB 2025-02-14 06:58:41,023 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26027.38 MB 2025-02-14 06:58:41,037 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 06:58:41,037 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 06:58:41,037 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:58:41,037 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:58:41,037 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22048.04 MB 2025-02-14 06:58:41,037 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23937.58 MB 2025-02-14 06:58:41,037 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 06:58:41,037 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28783.41 MB 2025-02-14 06:58:41,037 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28783.41 MB 2025-02-14 06:58:41,037 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:58:41,037 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25355.01 MB 2025-02-14 06:58:41,243 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 06:58:41,243 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 06:58:41,243 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 06:58:41,243 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:58:41,243 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23937.58 MB 2025-02-14 06:58:41,243 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26179.43 MB 2025-02-14 06:58:41,243 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 06:58:41,243 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28783.41 MB 2025-02-14 06:58:41,243 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34445.72 MB 2025-02-14 06:58:41,243 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 06:58:41,243 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31723.72 MB 2025-02-14 06:58:41,244 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 06:58:41,244 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 06:58:41,244 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 06:58:41,244 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:58:41,244 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22048.04 MB 2025-02-14 06:58:41,244 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26179.43 MB 2025-02-14 06:58:41,244 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 06:58:41,244 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28783.41 MB 2025-02-14 06:58:41,244 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34445.72 MB 2025-02-14 06:58:41,244 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 06:58:41,244 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31723.72 MB 2025-02-14 06:58:41,405 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 06:58:41,405 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 06:58:41,405 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 06:58:41,405 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:58:41,405 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27712.98 MB 2025-02-14 06:58:41,405 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28479.98 MB 2025-02-14 06:58:41,405 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 06:58:41,405 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34445.72 MB 2025-02-14 06:58:41,405 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34858.86 MB 2025-02-14 06:58:41,405 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 06:58:41,405 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29187.77 MB 2025-02-14 06:58:41,423 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 06:58:41,423 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 06:58:41,423 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:58:41,423 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:58:41,424 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28892.87 MB 2025-02-14 06:58:41,424 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29120.24 MB 2025-02-14 06:58:41,424 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.38 MB 2025-02-14 06:58:41,424 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34858.86 MB 2025-02-14 06:58:41,424 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34858.86 MB 2025-02-14 06:58:41,424 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:58:41,424 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29327.30 MB 2025-02-14 06:58:41,425 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 06:58:41,425 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 06:58:41,425 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.26 seconds 2025-02-14 06:58:41,425 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:58:41,425 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16815.13 MB 2025-02-14 06:58:41,425 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29320.36 MB 2025-02-14 06:58:41,425 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12505.23 MB 2025-02-14 06:58:41,425 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40584.09 MB 2025-02-14 06:58:41,425 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34858.86 MB 2025-02-14 06:58:41,425 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5725.22 MB 2025-02-14 06:58:41,425 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29327.30 MB 2025-02-14 06:58:41,691 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 06:58:41,691 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 06:58:41,691 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 06:58:41,691 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:58:41,691 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29320.36 MB 2025-02-14 06:58:41,691 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21804.66 MB 2025-02-14 06:58:41,691 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7515.69 MB 2025-02-14 06:58:41,691 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34858.86 MB 2025-02-14 06:58:41,691 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34858.86 MB 2025-02-14 06:58:41,691 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:58:41,691 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31820.04 MB 2025-02-14 06:58:41,709 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8123, cut from 8125 2025-02-14 06:58:41,709 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 06:58:41,715 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 06:58:41,715 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 06:58:41,715 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:58:41,715 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:58:41,715 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21804.66 MB 2025-02-14 06:58:41,715 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30204.05 MB 2025-02-14 06:58:41,715 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8399.39 MB 2025-02-14 06:58:41,715 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34858.86 MB 2025-02-14 06:58:41,715 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43209.72 MB 2025-02-14 06:58:41,715 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8350.86 MB 2025-02-14 06:58:41,715 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30204.05 MB 2025-02-14 06:58:41,876 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7915] 2025-02-14 06:58:41,878 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:58:41,878 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 06:58:41,879 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:58:41,879 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 06:58:41,884 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 06:58:41,886 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:58:41,886 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 06:58:41,886 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 06:59:28,820 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:59:28,820 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 06:59:28,825 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 06:59:28,829 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:59:28,829 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1566, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 06:59:28,830 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:59:28,830 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1566, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 06:59:52,960 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 06:59:52,960 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 06:59:52,960 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.12 seconds 2025-02-14 06:59:52,960 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:59:52,960 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23880.85 MB 2025-02-14 06:59:52,960 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29423.62 MB 2025-02-14 06:59:52,960 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5542.77 MB 2025-02-14 06:59:52,960 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51560.58 MB 2025-02-14 06:59:52,960 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39151.73 MB 2025-02-14 06:59:52,960 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12408.85 MB 2025-02-14 06:59:52,960 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38335.86 MB 2025-02-14 06:59:53,060 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 06:59:53,060 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 06:59:53,060 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 06:59:53,060 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:59:53,060 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29423.62 MB 2025-02-14 06:59:53,061 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23919.00 MB 2025-02-14 06:59:53,061 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5504.62 MB 2025-02-14 06:59:53,061 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39151.73 MB 2025-02-14 06:59:53,061 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51124.37 MB 2025-02-14 06:59:53,061 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 11972.64 MB 2025-02-14 06:59:53,061 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45623.25 MB 2025-02-14 06:59:54,988 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 06:59:54,988 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 06:59:54,988 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 06:59:54,988 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:59:54,988 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23919.00 MB 2025-02-14 06:59:54,988 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24449.84 MB 2025-02-14 06:59:54,988 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 06:59:54,988 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51124.37 MB 2025-02-14 06:59:54,988 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29433.53 MB 2025-02-14 06:59:54,988 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21690.84 MB 2025-02-14 06:59:54,988 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28429.17 MB 2025-02-14 06:59:55,004 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 06:59:55,004 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 06:59:55,004 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 06:59:55,004 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:59:55,004 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24449.84 MB 2025-02-14 06:59:55,004 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26339.37 MB 2025-02-14 06:59:55,004 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 06:59:55,004 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29433.53 MB 2025-02-14 06:59:55,004 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30377.25 MB 2025-02-14 06:59:55,004 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 06:59:55,004 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27756.80 MB 2025-02-14 06:59:55,219 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 06:59:55,219 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 06:59:55,219 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 06:59:55,219 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:59:55,219 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26339.37 MB 2025-02-14 06:59:55,219 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28581.23 MB 2025-02-14 06:59:55,219 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 06:59:55,219 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30377.25 MB 2025-02-14 06:59:55,219 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36511.42 MB 2025-02-14 06:59:55,219 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 06:59:55,219 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34125.51 MB 2025-02-14 06:59:55,220 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 06:59:55,220 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 06:59:55,220 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 06:59:55,220 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:59:55,220 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24449.84 MB 2025-02-14 06:59:55,220 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28581.23 MB 2025-02-14 06:59:55,220 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 06:59:55,220 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29433.53 MB 2025-02-14 06:59:55,220 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36511.42 MB 2025-02-14 06:59:55,220 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7077.89 MB 2025-02-14 06:59:55,220 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34125.51 MB 2025-02-14 06:59:55,400 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 06:59:55,400 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 06:59:55,400 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 06:59:55,400 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:59:55,400 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30114.77 MB 2025-02-14 06:59:55,400 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30881.77 MB 2025-02-14 06:59:55,400 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 06:59:55,400 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36511.42 MB 2025-02-14 06:59:55,400 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36928.75 MB 2025-02-14 06:59:55,400 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 06:59:55,400 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31589.56 MB 2025-02-14 06:59:55,419 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 06:59:55,419 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 06:59:55,419 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:59:55,419 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:59:55,419 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31294.66 MB 2025-02-14 06:59:55,419 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31523.79 MB 2025-02-14 06:59:55,419 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.13 MB 2025-02-14 06:59:55,419 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36928.75 MB 2025-02-14 06:59:55,419 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36928.75 MB 2025-02-14 06:59:55,419 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:59:55,419 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31754.56 MB 2025-02-14 06:59:55,420 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 06:59:55,420 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 06:59:55,421 - resource_logging.py:150 - __exit__ - DEBUG - Time: 26.59 seconds 2025-02-14 06:59:55,421 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:59:55,421 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18424.78 MB 2025-02-14 06:59:55,421 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31724.67 MB 2025-02-14 06:59:55,421 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13299.89 MB 2025-02-14 06:59:55,421 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51560.58 MB 2025-02-14 06:59:55,421 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36928.75 MB 2025-02-14 06:59:55,421 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14631.83 MB 2025-02-14 06:59:55,421 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31754.56 MB 2025-02-14 06:59:55,690 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 06:59:55,690 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 06:59:55,690 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 06:59:55,690 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:59:55,690 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31724.67 MB 2025-02-14 06:59:55,690 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23426.12 MB 2025-02-14 06:59:55,690 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8298.55 MB 2025-02-14 06:59:55,690 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36928.75 MB 2025-02-14 06:59:55,690 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36928.75 MB 2025-02-14 06:59:55,690 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 06:59:55,690 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34233.88 MB 2025-02-14 06:59:55,708 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8154, cut from 8156 2025-02-14 06:59:55,708 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-14 06:59:55,715 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 06:59:55,715 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 06:59:55,715 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 06:59:55,715 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 06:59:55,715 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23426.12 MB 2025-02-14 06:59:55,715 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31856.79 MB 2025-02-14 06:59:55,715 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8430.68 MB 2025-02-14 06:59:55,715 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36928.75 MB 2025-02-14 06:59:55,715 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45311.07 MB 2025-02-14 06:59:55,715 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8382.32 MB 2025-02-14 06:59:55,715 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31856.79 MB 2025-02-14 06:59:55,885 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7946] 2025-02-14 06:59:55,886 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:59:55,886 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 06:59:55,887 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:59:55,887 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 06:59:55,892 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 06:59:55,893 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 06:59:55,893 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 06:59:55,894 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-14 07:02:00,854 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:02:00,854 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 07:02:00,862 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 07:02:00,869 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:02:00,869 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 876, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 07:02:00,871 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:02:00,871 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 876, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 07:02:14,347 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 07:02:14,348 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 07:02:14,348 - resource_logging.py:150 - __exit__ - DEBUG - Time: 13.47 seconds 2025-02-14 07:02:14,348 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:02:14,348 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19072.82 MB 2025-02-14 07:02:14,348 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22172.93 MB 2025-02-14 07:02:14,348 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3100.11 MB 2025-02-14 07:02:14,348 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57883.49 MB 2025-02-14 07:02:14,348 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28361.88 MB 2025-02-14 07:02:14,348 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29521.61 MB 2025-02-14 07:02:14,348 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31036.41 MB 2025-02-14 07:02:14,419 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 07:02:14,419 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 07:02:14,419 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 07:02:14,419 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:02:14,419 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22172.93 MB 2025-02-14 07:02:14,419 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20331.90 MB 2025-02-14 07:02:14,419 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1841.03 MB 2025-02-14 07:02:14,419 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28361.88 MB 2025-02-14 07:02:14,419 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36618.37 MB 2025-02-14 07:02:14,419 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8256.49 MB 2025-02-14 07:02:14,419 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31862.48 MB 2025-02-14 07:02:16,346 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 07:02:16,346 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 07:02:16,346 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 07:02:16,346 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:02:16,346 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20331.90 MB 2025-02-14 07:02:16,346 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20862.74 MB 2025-02-14 07:02:16,346 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 07:02:16,346 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36618.37 MB 2025-02-14 07:02:16,346 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26675.77 MB 2025-02-14 07:02:16,346 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9942.60 MB 2025-02-14 07:02:16,346 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24842.08 MB 2025-02-14 07:02:16,359 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 07:02:16,359 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 07:02:16,359 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:02:16,359 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:02:16,359 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20862.74 MB 2025-02-14 07:02:16,360 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22752.28 MB 2025-02-14 07:02:16,360 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 07:02:16,360 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26675.77 MB 2025-02-14 07:02:16,360 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26675.77 MB 2025-02-14 07:02:16,360 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:02:16,360 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24169.71 MB 2025-02-14 07:02:16,577 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 07:02:16,577 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 07:02:16,577 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 07:02:16,577 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:02:16,577 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22752.28 MB 2025-02-14 07:02:16,577 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24994.13 MB 2025-02-14 07:02:16,577 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 07:02:16,577 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26675.77 MB 2025-02-14 07:02:16,577 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33281.80 MB 2025-02-14 07:02:16,577 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 07:02:16,577 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30538.42 MB 2025-02-14 07:02:16,578 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 07:02:16,578 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 07:02:16,578 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 07:02:16,578 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:02:16,578 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20862.74 MB 2025-02-14 07:02:16,578 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24994.13 MB 2025-02-14 07:02:16,578 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 07:02:16,578 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26675.77 MB 2025-02-14 07:02:16,578 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33281.80 MB 2025-02-14 07:02:16,578 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 07:02:16,578 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30538.42 MB 2025-02-14 07:02:16,753 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 07:02:16,753 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 07:02:16,753 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 07:02:16,753 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:02:16,753 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26527.68 MB 2025-02-14 07:02:16,753 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27294.68 MB 2025-02-14 07:02:16,753 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 07:02:16,753 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33281.80 MB 2025-02-14 07:02:16,753 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33697.04 MB 2025-02-14 07:02:16,753 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 07:02:16,753 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28002.47 MB 2025-02-14 07:02:16,773 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 07:02:16,773 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 07:02:16,773 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:02:16,773 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:02:16,773 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27707.57 MB 2025-02-14 07:02:16,773 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27936.16 MB 2025-02-14 07:02:16,773 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.59 MB 2025-02-14 07:02:16,773 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33697.04 MB 2025-02-14 07:02:16,773 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33697.04 MB 2025-02-14 07:02:16,773 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:02:16,773 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28137.74 MB 2025-02-14 07:02:16,774 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 07:02:16,774 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 07:02:16,774 - resource_logging.py:150 - __exit__ - DEBUG - Time: 15.90 seconds 2025-02-14 07:02:16,774 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:02:16,774 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16020.76 MB 2025-02-14 07:02:16,774 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28136.67 MB 2025-02-14 07:02:16,774 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12115.91 MB 2025-02-14 07:02:16,774 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57883.49 MB 2025-02-14 07:02:16,774 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33697.04 MB 2025-02-14 07:02:16,774 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24186.45 MB 2025-02-14 07:02:16,774 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28137.74 MB 2025-02-14 07:02:17,041 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 07:02:17,041 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 07:02:17,041 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 07:02:17,041 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:02:17,041 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28136.67 MB 2025-02-14 07:02:17,041 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21016.39 MB 2025-02-14 07:02:17,041 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7120.28 MB 2025-02-14 07:02:17,041 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33697.04 MB 2025-02-14 07:02:17,041 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33697.04 MB 2025-02-14 07:02:17,041 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:02:17,041 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30641.27 MB 2025-02-14 07:02:17,059 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8139, cut from 8141 2025-02-14 07:02:17,059 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 07:02:17,065 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 07:02:17,065 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 07:02:17,065 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:02:17,065 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:02:17,065 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21016.39 MB 2025-02-14 07:02:17,065 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29431.34 MB 2025-02-14 07:02:17,065 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8414.95 MB 2025-02-14 07:02:17,065 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33697.04 MB 2025-02-14 07:02:17,065 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42064.67 MB 2025-02-14 07:02:17,065 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8367.64 MB 2025-02-14 07:02:17,065 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29431.34 MB 2025-02-14 07:02:17,233 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7931] 2025-02-14 07:02:17,234 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:02:17,235 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 07:02:17,235 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:02:17,236 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 07:02:17,240 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 07:02:17,241 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:02:17,241 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 07:02:17,241 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 07:02:31,981 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:02:31,981 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 07:02:31,990 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 07:02:31,996 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:02:31,996 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2192, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 07:02:31,998 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:02:31,998 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2192, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 07:03:06,125 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 07:03:06,125 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 07:03:06,125 - resource_logging.py:150 - __exit__ - DEBUG - Time: 34.12 seconds 2025-02-14 07:03:06,125 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:03:06,125 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28242.91 MB 2025-02-14 07:03:06,125 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36000.28 MB 2025-02-14 07:03:06,125 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7757.37 MB 2025-02-14 07:03:06,125 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50432.31 MB 2025-02-14 07:03:06,125 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41385.20 MB 2025-02-14 07:03:06,125 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9047.11 MB 2025-02-14 07:03:06,125 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44962.85 MB 2025-02-14 07:03:06,264 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 07:03:06,265 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 07:03:06,265 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-14 07:03:06,265 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:03:06,265 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36000.28 MB 2025-02-14 07:03:06,265 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27173.37 MB 2025-02-14 07:03:06,265 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8826.90 MB 2025-02-14 07:03:06,265 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41385.20 MB 2025-02-14 07:03:06,265 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 69493.33 MB 2025-02-14 07:03:06,265 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 28108.13 MB 2025-02-14 07:03:06,265 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 58746.83 MB 2025-02-14 07:03:08,220 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 07:03:08,220 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 07:03:08,220 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.95 seconds 2025-02-14 07:03:08,220 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:03:08,220 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27173.37 MB 2025-02-14 07:03:08,220 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27704.22 MB 2025-02-14 07:03:08,220 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 07:03:08,220 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 69493.33 MB 2025-02-14 07:03:08,220 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30867.98 MB 2025-02-14 07:03:08,220 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -38625.35 MB 2025-02-14 07:03:08,220 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31684.59 MB 2025-02-14 07:03:08,235 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 07:03:08,235 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 07:03:08,235 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:03:08,235 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:03:08,235 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27704.22 MB 2025-02-14 07:03:08,235 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29593.75 MB 2025-02-14 07:03:08,235 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 07:03:08,235 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30867.98 MB 2025-02-14 07:03:08,235 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34170.99 MB 2025-02-14 07:03:08,235 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 07:03:08,235 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31011.18 MB 2025-02-14 07:03:08,448 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 07:03:08,448 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 07:03:08,448 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 07:03:08,448 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:03:08,448 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29593.75 MB 2025-02-14 07:03:08,448 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31835.61 MB 2025-02-14 07:03:08,448 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 07:03:08,448 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34170.99 MB 2025-02-14 07:03:08,448 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40305.16 MB 2025-02-14 07:03:08,448 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 07:03:08,448 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37379.89 MB 2025-02-14 07:03:08,448 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 07:03:08,448 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 07:03:08,448 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 07:03:08,449 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:03:08,449 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27704.22 MB 2025-02-14 07:03:08,449 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31835.61 MB 2025-02-14 07:03:08,449 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 07:03:08,449 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30867.98 MB 2025-02-14 07:03:08,449 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40305.16 MB 2025-02-14 07:03:08,449 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9437.18 MB 2025-02-14 07:03:08,449 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37379.89 MB 2025-02-14 07:03:08,617 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 07:03:08,617 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 07:03:08,617 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 07:03:08,617 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:03:08,617 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33369.15 MB 2025-02-14 07:03:08,617 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34136.15 MB 2025-02-14 07:03:08,617 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 07:03:08,617 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40305.16 MB 2025-02-14 07:03:08,617 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40720.40 MB 2025-02-14 07:03:08,617 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 07:03:08,617 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34843.94 MB 2025-02-14 07:03:08,636 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 07:03:08,636 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 07:03:08,636 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:03:08,636 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:03:08,636 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34549.04 MB 2025-02-14 07:03:08,636 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34777.94 MB 2025-02-14 07:03:08,636 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.90 MB 2025-02-14 07:03:08,636 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40720.40 MB 2025-02-14 07:03:08,636 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40720.40 MB 2025-02-14 07:03:08,636 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:03:08,636 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35012.12 MB 2025-02-14 07:03:08,638 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 07:03:08,638 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 07:03:08,638 - resource_logging.py:150 - __exit__ - DEBUG - Time: 36.64 seconds 2025-02-14 07:03:08,638 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:03:08,638 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20605.81 MB 2025-02-14 07:03:08,638 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34977.98 MB 2025-02-14 07:03:08,638 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14372.17 MB 2025-02-14 07:03:08,638 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50432.31 MB 2025-02-14 07:03:08,638 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40720.40 MB 2025-02-14 07:03:08,638 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9711.91 MB 2025-02-14 07:03:08,638 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35012.12 MB 2025-02-14 07:03:08,907 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 07:03:08,907 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 07:03:08,907 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 07:03:08,907 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:03:08,907 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34977.98 MB 2025-02-14 07:03:08,907 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25594.20 MB 2025-02-14 07:03:08,907 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9383.78 MB 2025-02-14 07:03:08,907 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40720.40 MB 2025-02-14 07:03:08,907 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40720.40 MB 2025-02-14 07:03:08,907 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:03:08,907 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37476.74 MB 2025-02-14 07:03:08,924 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8120, cut from 8122 2025-02-14 07:03:08,925 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 07:03:08,931 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 07:03:08,931 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 07:03:08,931 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:03:08,931 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:03:08,931 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25594.20 MB 2025-02-14 07:03:08,931 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33990.84 MB 2025-02-14 07:03:08,931 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8396.64 MB 2025-02-14 07:03:08,931 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40720.40 MB 2025-02-14 07:03:08,931 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49067.07 MB 2025-02-14 07:03:08,931 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8346.66 MB 2025-02-14 07:03:08,931 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33990.84 MB 2025-02-14 07:03:09,093 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7912] 2025-02-14 07:03:09,094 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:03:09,095 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 07:03:09,095 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:03:09,096 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 07:03:09,100 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 07:03:09,101 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:03:09,101 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 07:03:09,101 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 07:04:16,035 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:04:16,035 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 07:04:16,040 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 07:04:16,044 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:04:16,044 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 234, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 07:04:16,045 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:04:16,045 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 234, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 07:04:19,689 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 07:04:19,689 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 07:04:19,689 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.64 seconds 2025-02-14 07:04:19,689 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:04:19,689 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14599.90 MB 2025-02-14 07:04:19,689 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15428.02 MB 2025-02-14 07:04:19,689 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 828.11 MB 2025-02-14 07:04:19,689 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57413.73 MB 2025-02-14 07:04:19,689 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21091.06 MB 2025-02-14 07:04:19,689 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -36322.67 MB 2025-02-14 07:04:19,689 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24297.77 MB 2025-02-14 07:04:19,705 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 07:04:19,705 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 07:04:19,705 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:04:19,705 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:04:19,705 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15428.02 MB 2025-02-14 07:04:19,705 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15765.96 MB 2025-02-14 07:04:19,705 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 337.95 MB 2025-02-14 07:04:19,705 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21091.06 MB 2025-02-14 07:04:19,705 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21091.06 MB 2025-02-14 07:04:19,705 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:04:19,705 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18623.77 MB 2025-02-14 07:04:20,777 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 07:04:20,777 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 07:04:20,777 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.07 seconds 2025-02-14 07:04:20,777 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:04:20,777 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15765.96 MB 2025-02-14 07:04:20,777 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16064.56 MB 2025-02-14 07:04:20,777 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 298.60 MB 2025-02-14 07:04:20,777 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21091.06 MB 2025-02-14 07:04:20,777 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21091.06 MB 2025-02-14 07:04:20,777 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:04:20,777 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20021.33 MB 2025-02-14 07:04:20,786 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 07:04:20,786 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 07:04:20,786 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:04:20,786 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:04:20,786 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16064.56 MB 2025-02-14 07:04:20,786 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17127.17 MB 2025-02-14 07:04:20,786 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1062.60 MB 2025-02-14 07:04:20,786 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21091.06 MB 2025-02-14 07:04:20,786 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21091.06 MB 2025-02-14 07:04:20,786 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:04:20,786 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17924.47 MB 2025-02-14 07:04:20,909 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 07:04:20,909 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 07:04:20,909 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 07:04:20,909 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:04:20,909 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17127.17 MB 2025-02-14 07:04:20,909 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18388.24 MB 2025-02-14 07:04:20,909 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1261.07 MB 2025-02-14 07:04:20,909 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21091.06 MB 2025-02-14 07:04:20,909 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23221.76 MB 2025-02-14 07:04:20,909 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2130.71 MB 2025-02-14 07:04:20,909 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21506.87 MB 2025-02-14 07:04:20,909 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 07:04:20,909 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 07:04:20,909 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 07:04:20,909 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:04:20,909 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16064.56 MB 2025-02-14 07:04:20,909 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18388.24 MB 2025-02-14 07:04:20,910 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2323.68 MB 2025-02-14 07:04:20,910 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21091.06 MB 2025-02-14 07:04:20,910 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23221.76 MB 2025-02-14 07:04:20,910 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2130.71 MB 2025-02-14 07:04:20,910 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21506.87 MB 2025-02-14 07:04:21,008 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 07:04:21,008 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 07:04:21,008 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 07:04:21,008 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:04:21,008 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19250.86 MB 2025-02-14 07:04:21,008 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19682.30 MB 2025-02-14 07:04:21,008 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 431.44 MB 2025-02-14 07:04:21,008 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23221.76 MB 2025-02-14 07:04:21,008 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23454.55 MB 2025-02-14 07:04:21,008 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 232.78 MB 2025-02-14 07:04:21,008 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20080.43 MB 2025-02-14 07:04:21,020 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 07:04:21,020 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 07:04:21,020 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:04:21,020 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:04:21,020 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19914.55 MB 2025-02-14 07:04:21,020 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20139.58 MB 2025-02-14 07:04:21,020 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 225.03 MB 2025-02-14 07:04:21,020 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23454.55 MB 2025-02-14 07:04:21,020 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23456.65 MB 2025-02-14 07:04:21,020 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-14 07:04:21,020 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20199.01 MB 2025-02-14 07:04:21,021 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 07:04:21,021 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 07:04:21,021 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.97 seconds 2025-02-14 07:04:21,021 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:04:21,021 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13783.98 MB 2025-02-14 07:04:21,021 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20340.65 MB 2025-02-14 07:04:21,021 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6556.67 MB 2025-02-14 07:04:21,021 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57413.73 MB 2025-02-14 07:04:21,021 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23456.65 MB 2025-02-14 07:04:21,021 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33957.09 MB 2025-02-14 07:04:21,021 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20340.65 MB 2025-02-14 07:04:21,289 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 07:04:21,289 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 07:04:21,289 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 07:04:21,289 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:04:21,289 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14948.46 MB 2025-02-14 07:04:21,289 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17962.49 MB 2025-02-14 07:04:21,289 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-14 07:04:21,289 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23456.65 MB 2025-02-14 07:04:21,289 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23456.65 MB 2025-02-14 07:04:21,289 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:04:21,289 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18263.86 MB 2025-02-14 07:04:21,307 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 07:04:21,307 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 07:04:21,313 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 07:04:21,313 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 07:04:21,313 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:04:21,313 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:04:21,313 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17962.49 MB 2025-02-14 07:04:21,313 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26401.81 MB 2025-02-14 07:04:21,313 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.32 MB 2025-02-14 07:04:21,313 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23456.65 MB 2025-02-14 07:04:21,313 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31847.35 MB 2025-02-14 07:04:21,313 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 07:04:21,313 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26401.81 MB 2025-02-14 07:04:21,483 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 07:04:21,485 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:04:21,485 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 07:04:21,486 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:04:21,486 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 07:04:21,491 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 07:04:21,492 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:04:21,492 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 07:04:21,492 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 07:04:28,056 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:04:28,057 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 07:04:28,062 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 07:04:28,066 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:04:28,066 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1846, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 07:04:28,067 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:04:28,067 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1846, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 07:04:56,695 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 07:04:56,695 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 07:04:56,695 - resource_logging.py:150 - __exit__ - DEBUG - Time: 28.62 seconds 2025-02-14 07:04:56,695 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:04:56,695 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25831.93 MB 2025-02-14 07:04:56,695 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32364.82 MB 2025-02-14 07:04:56,695 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6532.89 MB 2025-02-14 07:04:56,695 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44432.36 MB 2025-02-14 07:04:56,695 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40208.70 MB 2025-02-14 07:04:56,695 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4223.66 MB 2025-02-14 07:04:56,695 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41192.91 MB 2025-02-14 07:04:56,766 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 07:04:56,766 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 07:04:56,766 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 07:04:56,766 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:04:56,766 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32364.82 MB 2025-02-14 07:04:56,766 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25374.63 MB 2025-02-14 07:04:56,766 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6990.19 MB 2025-02-14 07:04:56,766 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40208.70 MB 2025-02-14 07:04:56,766 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44728.06 MB 2025-02-14 07:04:56,766 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4519.36 MB 2025-02-14 07:04:56,766 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39960.95 MB 2025-02-14 07:04:58,679 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 07:04:58,679 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 07:04:58,679 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 07:04:58,679 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:04:58,679 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25374.63 MB 2025-02-14 07:04:58,679 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25905.47 MB 2025-02-14 07:04:58,679 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 07:04:58,679 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44728.06 MB 2025-02-14 07:04:58,679 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30895.24 MB 2025-02-14 07:04:58,679 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13832.81 MB 2025-02-14 07:04:58,679 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29884.80 MB 2025-02-14 07:04:58,694 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 07:04:58,694 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 07:04:58,694 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:04:58,694 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:04:58,694 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25905.47 MB 2025-02-14 07:04:58,694 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27795.00 MB 2025-02-14 07:04:58,694 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 07:04:58,694 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30895.24 MB 2025-02-14 07:04:58,694 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31838.96 MB 2025-02-14 07:04:58,694 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 07:04:58,694 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29212.43 MB 2025-02-14 07:04:58,904 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 07:04:58,904 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 07:04:58,904 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 07:04:58,904 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:04:58,904 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27795.00 MB 2025-02-14 07:04:58,904 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30036.86 MB 2025-02-14 07:04:58,904 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 07:04:58,904 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31838.96 MB 2025-02-14 07:04:58,904 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37973.13 MB 2025-02-14 07:04:58,904 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 07:04:58,904 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35581.14 MB 2025-02-14 07:04:58,905 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 07:04:58,905 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 07:04:58,905 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 07:04:58,905 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:04:58,905 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25905.47 MB 2025-02-14 07:04:58,905 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30036.86 MB 2025-02-14 07:04:58,905 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 07:04:58,905 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30895.24 MB 2025-02-14 07:04:58,905 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37973.13 MB 2025-02-14 07:04:58,905 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7077.89 MB 2025-02-14 07:04:58,905 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35581.14 MB 2025-02-14 07:04:59,071 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 07:04:59,071 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 07:04:59,071 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 07:04:59,071 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:04:59,071 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31570.40 MB 2025-02-14 07:04:59,071 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32337.40 MB 2025-02-14 07:04:59,071 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 07:04:59,071 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37973.13 MB 2025-02-14 07:04:59,071 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38390.46 MB 2025-02-14 07:04:59,071 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 07:04:59,071 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33045.19 MB 2025-02-14 07:04:59,090 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 07:04:59,090 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 07:04:59,090 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:04:59,090 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:04:59,090 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32750.29 MB 2025-02-14 07:04:59,090 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32985.16 MB 2025-02-14 07:04:59,090 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 234.87 MB 2025-02-14 07:04:59,090 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38390.46 MB 2025-02-14 07:04:59,090 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38390.46 MB 2025-02-14 07:04:59,090 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:04:59,090 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33171.34 MB 2025-02-14 07:04:59,091 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 07:04:59,092 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 07:04:59,092 - resource_logging.py:150 - __exit__ - DEBUG - Time: 31.02 seconds 2025-02-14 07:04:59,092 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:04:59,092 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19400.32 MB 2025-02-14 07:04:59,092 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33186.23 MB 2025-02-14 07:04:59,092 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13785.92 MB 2025-02-14 07:04:59,092 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44432.36 MB 2025-02-14 07:04:59,092 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38390.46 MB 2025-02-14 07:04:59,092 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6041.89 MB 2025-02-14 07:04:59,092 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33186.23 MB 2025-02-14 07:04:59,361 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 07:04:59,361 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 07:04:59,361 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 07:04:59,362 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:04:59,362 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33186.23 MB 2025-02-14 07:04:59,362 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24404.71 MB 2025-02-14 07:04:59,362 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8781.53 MB 2025-02-14 07:04:59,362 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38390.46 MB 2025-02-14 07:04:59,362 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38390.46 MB 2025-02-14 07:04:59,362 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:04:59,362 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35697.90 MB 2025-02-14 07:04:59,380 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 07:04:59,380 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 07:04:59,386 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 07:04:59,386 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 07:04:59,386 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:04:59,386 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:04:59,386 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24404.71 MB 2025-02-14 07:04:59,386 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32843.73 MB 2025-02-14 07:04:59,386 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 07:04:59,386 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38390.46 MB 2025-02-14 07:04:59,386 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46781.17 MB 2025-02-14 07:04:59,386 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 07:04:59,386 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32843.73 MB 2025-02-14 07:04:59,549 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 07:04:59,550 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:04:59,550 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 07:04:59,551 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:04:59,551 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 07:04:59,556 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 07:04:59,557 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:04:59,557 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 07:04:59,557 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 07:05:10,845 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:05:10,845 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 07:05:10,850 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 07:05:10,854 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:05:10,854 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 115, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 07:05:10,855 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:05:10,855 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 115, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 07:05:12,665 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 07:05:12,665 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 07:05:12,665 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.81 seconds 2025-02-14 07:05:12,665 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:05:12,665 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13770.05 MB 2025-02-14 07:05:12,666 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14177.02 MB 2025-02-14 07:05:12,666 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 406.98 MB 2025-02-14 07:05:12,666 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59366.18 MB 2025-02-14 07:05:12,666 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17853.05 MB 2025-02-14 07:05:12,666 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -41513.12 MB 2025-02-14 07:05:12,666 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23014.92 MB 2025-02-14 07:05:12,668 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 07:05:12,668 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 07:05:12,668 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 07:05:12,668 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:05:12,669 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14177.02 MB 2025-02-14 07:05:12,669 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14374.20 MB 2025-02-14 07:05:12,669 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 197.18 MB 2025-02-14 07:05:12,669 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17853.05 MB 2025-02-14 07:05:12,669 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17853.05 MB 2025-02-14 07:05:12,669 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:05:12,669 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14984.74 MB 2025-02-14 07:05:13,234 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 07:05:13,234 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 07:05:13,234 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.56 seconds 2025-02-14 07:05:13,234 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:05:13,234 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14374.20 MB 2025-02-14 07:05:13,234 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14526.82 MB 2025-02-14 07:05:13,234 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 152.62 MB 2025-02-14 07:05:13,234 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17853.05 MB 2025-02-14 07:05:13,234 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17853.05 MB 2025-02-14 07:05:13,234 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:05:13,234 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18460.74 MB 2025-02-14 07:05:13,240 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 07:05:13,240 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 07:05:13,240 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 07:05:13,240 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:05:13,240 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14526.76 MB 2025-02-14 07:05:13,240 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15069.86 MB 2025-02-14 07:05:13,240 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 543.11 MB 2025-02-14 07:05:13,240 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17853.05 MB 2025-02-14 07:05:13,240 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17853.05 MB 2025-02-14 07:05:13,240 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:05:13,240 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15477.38 MB 2025-02-14 07:05:13,356 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 07:05:13,356 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 07:05:13,356 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 07:05:13,356 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:05:13,356 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15069.86 MB 2025-02-14 07:05:13,356 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15729.52 MB 2025-02-14 07:05:13,356 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 659.65 MB 2025-02-14 07:05:13,356 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17853.05 MB 2025-02-14 07:05:13,356 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18398.31 MB 2025-02-14 07:05:13,356 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 545.26 MB 2025-02-14 07:05:13,356 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17308.38 MB 2025-02-14 07:05:13,357 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 07:05:13,357 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 07:05:13,357 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 07:05:13,357 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:05:13,357 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14526.76 MB 2025-02-14 07:05:13,357 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15729.52 MB 2025-02-14 07:05:13,357 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1202.76 MB 2025-02-14 07:05:13,357 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17853.05 MB 2025-02-14 07:05:13,357 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18398.31 MB 2025-02-14 07:05:13,357 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 545.26 MB 2025-02-14 07:05:13,357 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17308.38 MB 2025-02-14 07:05:13,416 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 07:05:13,416 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 07:05:13,416 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 07:05:13,416 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:05:13,416 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16366.48 MB 2025-02-14 07:05:13,416 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16643.52 MB 2025-02-14 07:05:13,416 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 277.04 MB 2025-02-14 07:05:13,416 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18398.31 MB 2025-02-14 07:05:13,416 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18574.48 MB 2025-02-14 07:05:13,416 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 176.16 MB 2025-02-14 07:05:13,416 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16847.01 MB 2025-02-14 07:05:13,423 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 07:05:13,423 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 07:05:13,423 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 07:05:13,423 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:05:13,423 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16818.76 MB 2025-02-14 07:05:13,423 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17048.37 MB 2025-02-14 07:05:13,423 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.61 MB 2025-02-14 07:05:13,423 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18574.48 MB 2025-02-14 07:05:13,423 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18574.48 MB 2025-02-14 07:05:13,423 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:05:13,423 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17048.37 MB 2025-02-14 07:05:13,424 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 07:05:13,424 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 07:05:13,424 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.57 seconds 2025-02-14 07:05:13,424 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:05:13,424 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13369.38 MB 2025-02-14 07:05:13,424 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17248.48 MB 2025-02-14 07:05:13,424 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3879.11 MB 2025-02-14 07:05:13,424 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59366.18 MB 2025-02-14 07:05:13,424 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18574.48 MB 2025-02-14 07:05:13,424 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -40791.70 MB 2025-02-14 07:05:13,424 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17248.48 MB 2025-02-14 07:05:13,696 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 07:05:13,696 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 07:05:13,696 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 07:05:13,696 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:05:13,696 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17248.48 MB 2025-02-14 07:05:13,696 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20248.14 MB 2025-02-14 07:05:13,697 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2999.66 MB 2025-02-14 07:05:13,697 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18574.48 MB 2025-02-14 07:05:13,697 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21929.92 MB 2025-02-14 07:05:13,697 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3355.44 MB 2025-02-14 07:05:13,697 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20548.97 MB 2025-02-14 07:05:13,714 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8123, cut from 8125 2025-02-14 07:05:13,715 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 07:05:13,721 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 07:05:13,721 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 07:05:13,721 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:05:13,721 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:05:13,721 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20248.14 MB 2025-02-14 07:05:13,721 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28647.53 MB 2025-02-14 07:05:13,721 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8399.39 MB 2025-02-14 07:05:13,721 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21929.92 MB 2025-02-14 07:05:13,721 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32369.54 MB 2025-02-14 07:05:13,721 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10439.62 MB 2025-02-14 07:05:13,721 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28647.53 MB 2025-02-14 07:05:13,884 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7915] 2025-02-14 07:05:13,886 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:05:13,886 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 07:05:13,887 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:05:13,887 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 07:05:13,891 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 07:05:13,893 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:05:13,893 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 07:05:13,893 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 07:06:41,885 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:06:41,885 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 07:06:41,890 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 07:06:41,894 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:06:41,894 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 204, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 07:06:41,895 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:06:41,895 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 204, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 07:06:45,062 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 07:06:45,062 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 07:06:45,062 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.16 seconds 2025-02-14 07:06:45,062 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:06:45,063 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22726.35 MB 2025-02-14 07:06:45,063 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23448.29 MB 2025-02-14 07:06:45,063 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 721.94 MB 2025-02-14 07:06:45,063 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40722.50 MB 2025-02-14 07:06:45,063 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25562.19 MB 2025-02-14 07:06:45,063 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15160.31 MB 2025-02-14 07:06:45,063 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32425.02 MB 2025-02-14 07:06:45,077 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 07:06:45,077 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 07:06:45,077 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:06:45,077 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:06:45,077 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23448.29 MB 2025-02-14 07:06:45,077 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23721.90 MB 2025-02-14 07:06:45,077 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 273.61 MB 2025-02-14 07:06:45,077 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25562.19 MB 2025-02-14 07:06:45,077 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27592.23 MB 2025-02-14 07:06:45,077 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2030.04 MB 2025-02-14 07:06:45,077 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26203.88 MB 2025-02-14 07:06:46,012 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 07:06:46,012 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 07:06:46,012 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.93 seconds 2025-02-14 07:06:46,012 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:06:46,012 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23721.90 MB 2025-02-14 07:06:46,012 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23978.03 MB 2025-02-14 07:06:46,012 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 256.13 MB 2025-02-14 07:06:46,012 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27592.23 MB 2025-02-14 07:06:46,012 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25853.69 MB 2025-02-14 07:06:46,012 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1738.54 MB 2025-02-14 07:06:46,012 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27978.56 MB 2025-02-14 07:06:46,021 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 07:06:46,021 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 07:06:46,021 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:06:46,021 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:06:46,021 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23977.97 MB 2025-02-14 07:06:46,021 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24889.45 MB 2025-02-14 07:06:46,021 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 911.48 MB 2025-02-14 07:06:46,021 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25853.69 MB 2025-02-14 07:06:46,021 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27227.32 MB 2025-02-14 07:06:46,021 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1373.63 MB 2025-02-14 07:06:46,021 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25573.36 MB 2025-02-14 07:06:46,124 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 07:06:46,124 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 07:06:46,124 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 07:06:46,124 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:06:46,124 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24889.45 MB 2025-02-14 07:06:46,124 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25971.18 MB 2025-02-14 07:06:46,124 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1081.73 MB 2025-02-14 07:06:46,124 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27227.32 MB 2025-02-14 07:06:46,124 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29970.40 MB 2025-02-14 07:06:46,124 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2743.07 MB 2025-02-14 07:06:46,124 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28648.09 MB 2025-02-14 07:06:46,125 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 07:06:46,125 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 07:06:46,125 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 07:06:46,125 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:06:46,125 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23977.97 MB 2025-02-14 07:06:46,125 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25971.18 MB 2025-02-14 07:06:46,125 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1993.21 MB 2025-02-14 07:06:46,125 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25853.69 MB 2025-02-14 07:06:46,125 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29970.40 MB 2025-02-14 07:06:46,125 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4116.71 MB 2025-02-14 07:06:46,125 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28648.09 MB 2025-02-14 07:06:46,203 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 07:06:46,203 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 07:06:46,203 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 07:06:46,203 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:06:46,203 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26711.11 MB 2025-02-14 07:06:46,203 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27083.02 MB 2025-02-14 07:06:46,203 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 371.91 MB 2025-02-14 07:06:46,203 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29970.40 MB 2025-02-14 07:06:46,203 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30171.73 MB 2025-02-14 07:06:46,203 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 201.33 MB 2025-02-14 07:06:46,203 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27429.96 MB 2025-02-14 07:06:46,213 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 07:06:46,213 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 07:06:46,213 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:06:46,213 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:06:46,213 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27282.25 MB 2025-02-14 07:06:46,213 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27509.40 MB 2025-02-14 07:06:46,213 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.15 MB 2025-02-14 07:06:46,213 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30171.73 MB 2025-02-14 07:06:46,213 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30171.73 MB 2025-02-14 07:06:46,213 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:06:46,213 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27560.46 MB 2025-02-14 07:06:46,215 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 07:06:46,215 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 07:06:46,215 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.32 seconds 2025-02-14 07:06:46,215 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:06:46,215 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22015.59 MB 2025-02-14 07:06:46,215 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27710.48 MB 2025-02-14 07:06:46,215 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5694.88 MB 2025-02-14 07:06:46,215 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40722.50 MB 2025-02-14 07:06:46,215 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30171.73 MB 2025-02-14 07:06:46,215 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10550.77 MB 2025-02-14 07:06:46,215 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27710.48 MB 2025-02-14 07:06:46,482 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 07:06:46,482 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 07:06:46,482 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 07:06:46,482 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:06:46,482 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27710.48 MB 2025-02-14 07:06:46,482 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30724.51 MB 2025-02-14 07:06:46,482 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-14 07:06:46,482 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30171.73 MB 2025-02-14 07:06:46,482 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32319.21 MB 2025-02-14 07:06:46,482 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2147.48 MB 2025-02-14 07:06:46,482 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31026.14 MB 2025-02-14 07:06:46,500 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 07:06:46,500 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 07:06:46,506 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 07:06:46,507 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 07:06:46,507 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:06:46,507 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:06:46,507 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30724.51 MB 2025-02-14 07:06:46,507 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39163.53 MB 2025-02-14 07:06:46,507 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 07:06:46,507 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32319.21 MB 2025-02-14 07:06:46,507 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42809.16 MB 2025-02-14 07:06:46,507 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 07:06:46,507 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39163.53 MB 2025-02-14 07:06:46,658 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 07:06:46,660 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:06:46,660 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 07:06:46,661 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:06:46,661 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 07:06:46,665 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 07:06:46,666 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:06:46,666 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 07:06:46,666 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 07:07:56,365 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:07:56,365 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 07:07:56,370 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 07:07:56,375 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:07:56,375 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2057, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 07:07:56,376 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:07:56,376 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2057, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 07:08:28,060 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 07:08:28,060 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 07:08:28,060 - resource_logging.py:150 - __exit__ - DEBUG - Time: 31.68 seconds 2025-02-14 07:08:28,060 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:08:28,060 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35638.35 MB 2025-02-14 07:08:28,060 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42917.95 MB 2025-02-14 07:08:28,060 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7279.61 MB 2025-02-14 07:08:28,060 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55396.27 MB 2025-02-14 07:08:28,060 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46810.53 MB 2025-02-14 07:08:28,060 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8585.74 MB 2025-02-14 07:08:28,060 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51905.30 MB 2025-02-14 07:08:28,193 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 07:08:28,193 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 07:08:28,193 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 07:08:28,194 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:08:28,194 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42917.95 MB 2025-02-14 07:08:28,194 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34808.73 MB 2025-02-14 07:08:28,194 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8109.22 MB 2025-02-14 07:08:28,194 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46810.53 MB 2025-02-14 07:08:28,194 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 77185.68 MB 2025-02-14 07:08:28,194 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 30375.15 MB 2025-02-14 07:08:28,194 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 64507.70 MB 2025-02-14 07:08:30,150 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 07:08:30,150 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 07:08:30,150 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.95 seconds 2025-02-14 07:08:30,150 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:08:30,150 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34808.73 MB 2025-02-14 07:08:30,150 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35339.58 MB 2025-02-14 07:08:30,150 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 07:08:30,150 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 77185.68 MB 2025-02-14 07:08:30,150 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38168.17 MB 2025-02-14 07:08:30,150 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -39017.51 MB 2025-02-14 07:08:30,150 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39319.16 MB 2025-02-14 07:08:30,164 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 07:08:30,164 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 07:08:30,164 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:08:30,164 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:08:30,165 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35339.58 MB 2025-02-14 07:08:30,165 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37229.11 MB 2025-02-14 07:08:30,165 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 07:08:30,165 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38168.17 MB 2025-02-14 07:08:30,165 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41001.42 MB 2025-02-14 07:08:30,165 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2833.25 MB 2025-02-14 07:08:30,165 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38646.54 MB 2025-02-14 07:08:30,369 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 07:08:30,369 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 07:08:30,369 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 07:08:30,369 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:08:30,369 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37229.11 MB 2025-02-14 07:08:30,369 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39470.97 MB 2025-02-14 07:08:30,369 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 07:08:30,369 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41001.42 MB 2025-02-14 07:08:30,369 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47135.59 MB 2025-02-14 07:08:30,369 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 07:08:30,369 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45015.25 MB 2025-02-14 07:08:30,370 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 07:08:30,370 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 07:08:30,370 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 07:08:30,370 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:08:30,370 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35339.58 MB 2025-02-14 07:08:30,370 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39470.97 MB 2025-02-14 07:08:30,370 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 07:08:30,370 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38168.17 MB 2025-02-14 07:08:30,370 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47135.59 MB 2025-02-14 07:08:30,370 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8967.42 MB 2025-02-14 07:08:30,370 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45015.25 MB 2025-02-14 07:08:30,538 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 07:08:30,539 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 07:08:30,539 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 07:08:30,539 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:08:30,539 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41004.51 MB 2025-02-14 07:08:30,539 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41771.51 MB 2025-02-14 07:08:30,539 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 07:08:30,539 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47135.59 MB 2025-02-14 07:08:30,539 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47552.92 MB 2025-02-14 07:08:30,539 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 07:08:30,539 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42479.30 MB 2025-02-14 07:08:30,558 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 07:08:30,558 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 07:08:30,558 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:08:30,558 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:08:30,558 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42184.40 MB 2025-02-14 07:08:30,558 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42412.75 MB 2025-02-14 07:08:30,558 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.35 MB 2025-02-14 07:08:30,558 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47552.92 MB 2025-02-14 07:08:30,558 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47552.92 MB 2025-02-14 07:08:30,558 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:08:30,558 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42652.50 MB 2025-02-14 07:08:30,559 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 07:08:30,559 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 07:08:30,559 - resource_logging.py:150 - __exit__ - DEBUG - Time: 34.18 seconds 2025-02-14 07:08:30,559 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:08:30,559 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28471.59 MB 2025-02-14 07:08:30,559 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42613.48 MB 2025-02-14 07:08:30,559 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14141.89 MB 2025-02-14 07:08:30,559 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55396.27 MB 2025-02-14 07:08:30,559 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47552.92 MB 2025-02-14 07:08:30,559 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7843.35 MB 2025-02-14 07:08:30,559 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42652.50 MB 2025-02-14 07:08:30,829 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 07:08:30,829 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 07:08:30,829 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 07:08:30,829 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:08:30,829 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42613.48 MB 2025-02-14 07:08:30,829 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33470.35 MB 2025-02-14 07:08:30,829 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9143.12 MB 2025-02-14 07:08:30,829 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47552.92 MB 2025-02-14 07:08:30,829 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47552.92 MB 2025-02-14 07:08:30,829 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:08:30,829 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45120.84 MB 2025-02-14 07:08:30,847 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8148, cut from 8150 2025-02-14 07:08:30,847 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 07:08:30,853 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 07:08:30,853 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 07:08:30,853 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:08:30,853 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:08:30,853 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33470.35 MB 2025-02-14 07:08:30,853 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41895.30 MB 2025-02-14 07:08:30,853 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8424.95 MB 2025-02-14 07:08:30,853 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47552.92 MB 2025-02-14 07:08:30,853 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55928.95 MB 2025-02-14 07:08:30,853 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8376.03 MB 2025-02-14 07:08:30,854 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41895.30 MB 2025-02-14 07:08:31,003 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7940] 2025-02-14 07:08:31,004 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:08:31,004 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 07:08:31,005 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:08:31,005 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 07:08:31,010 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 07:08:31,011 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:08:31,011 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 07:08:31,011 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 07:09:21,219 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:09:21,219 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 07:09:21,224 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 07:09:21,228 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:09:21,228 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1801, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 07:09:21,229 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:09:21,229 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1801, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 07:09:49,130 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 07:09:49,130 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 07:09:49,130 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.89 seconds 2025-02-14 07:09:49,130 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:09:49,130 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33854.50 MB 2025-02-14 07:09:49,130 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40228.13 MB 2025-02-14 07:09:49,130 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6373.64 MB 2025-02-14 07:09:49,130 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64304.97 MB 2025-02-14 07:09:49,130 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45885.69 MB 2025-02-14 07:09:49,130 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18419.29 MB 2025-02-14 07:09:49,130 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49215.48 MB 2025-02-14 07:09:49,242 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 07:09:49,242 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 07:09:49,242 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 07:09:49,242 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:09:49,242 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40228.13 MB 2025-02-14 07:09:49,242 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33476.82 MB 2025-02-14 07:09:49,242 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6751.31 MB 2025-02-14 07:09:49,242 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45885.69 MB 2025-02-14 07:09:49,242 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 67469.57 MB 2025-02-14 07:09:49,242 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 21583.89 MB 2025-02-14 07:09:49,242 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 58335.96 MB 2025-02-14 07:09:51,189 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 07:09:51,189 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 07:09:51,189 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.95 seconds 2025-02-14 07:09:51,189 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:09:51,189 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33476.82 MB 2025-02-14 07:09:51,189 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34007.66 MB 2025-02-14 07:09:51,189 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 07:09:51,189 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 67469.57 MB 2025-02-14 07:09:51,189 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40925.92 MB 2025-02-14 07:09:51,189 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26543.65 MB 2025-02-14 07:09:51,189 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37986.21 MB 2025-02-14 07:09:51,203 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 07:09:51,203 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 07:09:51,203 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:09:51,203 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:09:51,203 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34007.66 MB 2025-02-14 07:09:51,203 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35897.20 MB 2025-02-14 07:09:51,203 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 07:09:51,203 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40925.92 MB 2025-02-14 07:09:51,203 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40928.02 MB 2025-02-14 07:09:51,203 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-14 07:09:51,203 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37314.63 MB 2025-02-14 07:09:51,414 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 07:09:51,414 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 07:09:51,414 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 07:09:51,414 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:09:51,414 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35897.20 MB 2025-02-14 07:09:51,414 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38139.05 MB 2025-02-14 07:09:51,414 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 07:09:51,414 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40928.02 MB 2025-02-14 07:09:51,414 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46118.47 MB 2025-02-14 07:09:51,414 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 07:09:51,414 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43683.34 MB 2025-02-14 07:09:51,414 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 07:09:51,414 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 07:09:51,414 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 07:09:51,414 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:09:51,414 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34007.66 MB 2025-02-14 07:09:51,414 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38139.05 MB 2025-02-14 07:09:51,415 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 07:09:51,415 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40925.92 MB 2025-02-14 07:09:51,415 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46118.47 MB 2025-02-14 07:09:51,415 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5192.55 MB 2025-02-14 07:09:51,415 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43683.34 MB 2025-02-14 07:09:51,585 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 07:09:51,585 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 07:09:51,585 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 07:09:51,585 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:09:51,585 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39672.60 MB 2025-02-14 07:09:51,585 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40439.60 MB 2025-02-14 07:09:51,585 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 07:09:51,585 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46118.47 MB 2025-02-14 07:09:51,585 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46535.80 MB 2025-02-14 07:09:51,585 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 07:09:51,585 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41147.39 MB 2025-02-14 07:09:51,604 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 07:09:51,604 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 07:09:51,604 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:09:51,604 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:09:51,604 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40852.49 MB 2025-02-14 07:09:51,604 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41081.29 MB 2025-02-14 07:09:51,604 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.80 MB 2025-02-14 07:09:51,604 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46535.80 MB 2025-02-14 07:09:51,604 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46535.80 MB 2025-02-14 07:09:51,604 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:09:51,604 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41276.81 MB 2025-02-14 07:09:51,605 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 07:09:51,605 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 07:09:51,605 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.37 seconds 2025-02-14 07:09:51,605 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:09:51,605 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27579.67 MB 2025-02-14 07:09:51,605 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41281.67 MB 2025-02-14 07:09:51,605 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13702.00 MB 2025-02-14 07:09:51,605 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64304.97 MB 2025-02-14 07:09:51,605 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46535.80 MB 2025-02-14 07:09:51,605 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17769.17 MB 2025-02-14 07:09:51,605 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41281.67 MB 2025-02-14 07:09:51,876 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 07:09:51,876 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 07:09:51,876 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 07:09:51,876 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:09:51,876 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41281.67 MB 2025-02-14 07:09:51,876 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32573.27 MB 2025-02-14 07:09:51,876 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8708.40 MB 2025-02-14 07:09:51,876 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46535.80 MB 2025-02-14 07:09:51,876 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46535.80 MB 2025-02-14 07:09:51,876 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:09:51,876 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43784.74 MB 2025-02-14 07:09:51,894 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8134, cut from 8136 2025-02-14 07:09:51,894 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 07:09:51,901 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 07:09:51,901 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 07:09:51,901 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:09:51,901 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:09:51,901 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32573.27 MB 2025-02-14 07:09:51,901 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40983.07 MB 2025-02-14 07:09:51,901 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8409.81 MB 2025-02-14 07:09:51,901 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46535.80 MB 2025-02-14 07:09:51,901 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54897.15 MB 2025-02-14 07:09:51,901 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8361.35 MB 2025-02-14 07:09:51,901 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40983.07 MB 2025-02-14 07:09:52,063 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7926] 2025-02-14 07:09:52,064 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:09:52,064 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 07:09:52,065 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:09:52,065 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 07:09:52,070 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 07:09:52,071 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:09:52,071 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 07:09:52,071 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 07:11:05,580 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:11:05,580 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 07:11:05,585 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 07:11:05,589 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:11:05,589 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1217, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 07:11:05,590 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:11:05,590 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1217, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 07:11:24,321 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 07:11:24,321 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 07:11:24,322 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.72 seconds 2025-02-14 07:11:24,322 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:11:24,322 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29785.09 MB 2025-02-14 07:11:24,322 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34092.64 MB 2025-02-14 07:11:24,322 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4307.55 MB 2025-02-14 07:11:24,322 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 67438.12 MB 2025-02-14 07:11:24,322 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39596.33 MB 2025-02-14 07:11:24,322 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27841.79 MB 2025-02-14 07:11:24,322 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43106.83 MB 2025-02-14 07:11:24,364 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 07:11:24,364 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 07:11:24,364 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-14 07:11:24,364 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:11:24,364 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34092.64 MB 2025-02-14 07:11:24,364 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29303.06 MB 2025-02-14 07:11:24,364 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4789.58 MB 2025-02-14 07:11:24,364 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39596.33 MB 2025-02-14 07:11:24,364 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39596.33 MB 2025-02-14 07:11:24,364 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:11:24,364 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37439.14 MB 2025-02-14 07:11:25,498 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 07:11:25,498 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 07:11:25,498 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.13 seconds 2025-02-14 07:11:25,498 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:11:25,498 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29303.06 MB 2025-02-14 07:11:25,498 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29618.91 MB 2025-02-14 07:11:25,498 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 315.85 MB 2025-02-14 07:11:25,498 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39596.33 MB 2025-02-14 07:11:25,498 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35288.78 MB 2025-02-14 07:11:25,498 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4307.55 MB 2025-02-14 07:11:25,498 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33557.64 MB 2025-02-14 07:11:25,507 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 07:11:25,507 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 07:11:25,507 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:11:25,507 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:11:25,507 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29618.91 MB 2025-02-14 07:11:25,507 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30742.91 MB 2025-02-14 07:11:25,507 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1124.00 MB 2025-02-14 07:11:25,507 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35288.78 MB 2025-02-14 07:11:25,507 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35290.87 MB 2025-02-14 07:11:25,507 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-14 07:11:25,507 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31586.28 MB 2025-02-14 07:11:25,631 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 07:11:25,631 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 07:11:25,631 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 07:11:25,631 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:11:25,631 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30742.91 MB 2025-02-14 07:11:25,631 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32076.84 MB 2025-02-14 07:11:25,631 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1333.93 MB 2025-02-14 07:11:25,631 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35290.87 MB 2025-02-14 07:11:25,631 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36976.98 MB 2025-02-14 07:11:25,631 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1686.11 MB 2025-02-14 07:11:25,631 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35377.23 MB 2025-02-14 07:11:25,632 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 07:11:25,632 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 07:11:25,632 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 07:11:25,632 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:11:25,632 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29618.91 MB 2025-02-14 07:11:25,632 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32076.84 MB 2025-02-14 07:11:25,632 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2457.93 MB 2025-02-14 07:11:25,632 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35288.78 MB 2025-02-14 07:11:25,632 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36976.98 MB 2025-02-14 07:11:25,632 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1688.21 MB 2025-02-14 07:11:25,632 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35377.23 MB 2025-02-14 07:11:25,729 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 07:11:25,729 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 07:11:25,729 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 07:11:25,729 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:11:25,729 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32989.30 MB 2025-02-14 07:11:25,729 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33446.19 MB 2025-02-14 07:11:25,729 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 456.89 MB 2025-02-14 07:11:25,729 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36976.98 MB 2025-02-14 07:11:25,729 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37226.55 MB 2025-02-14 07:11:25,729 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 249.56 MB 2025-02-14 07:11:25,729 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33867.32 MB 2025-02-14 07:11:25,741 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 07:11:25,741 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 07:11:25,741 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:11:25,741 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:11:25,741 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33691.86 MB 2025-02-14 07:11:25,741 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33911.83 MB 2025-02-14 07:11:25,741 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 219.97 MB 2025-02-14 07:11:25,741 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37226.55 MB 2025-02-14 07:11:25,741 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37226.55 MB 2025-02-14 07:11:25,741 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:11:25,741 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33968.88 MB 2025-02-14 07:11:25,743 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 07:11:25,743 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 07:11:25,743 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.15 seconds 2025-02-14 07:11:25,743 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:11:25,743 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25544.97 MB 2025-02-14 07:11:25,743 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34112.90 MB 2025-02-14 07:11:25,743 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8567.94 MB 2025-02-14 07:11:25,743 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 67438.12 MB 2025-02-14 07:11:25,743 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37226.55 MB 2025-02-14 07:11:25,743 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30211.57 MB 2025-02-14 07:11:25,743 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34112.90 MB 2025-02-14 07:11:26,031 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 07:11:26,031 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 07:11:26,031 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-14 07:11:26,031 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:11:26,031 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34112.90 MB 2025-02-14 07:11:26,031 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37126.93 MB 2025-02-14 07:11:26,031 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-14 07:11:26,031 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37226.55 MB 2025-02-14 07:11:26,031 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38568.72 MB 2025-02-14 07:11:26,031 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1342.18 MB 2025-02-14 07:11:26,031 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37428.56 MB 2025-02-14 07:11:26,049 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 07:11:26,049 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 07:11:26,055 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 07:11:26,055 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 07:11:26,055 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:11:26,055 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:11:26,055 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29784.89 MB 2025-02-14 07:11:26,055 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38223.91 MB 2025-02-14 07:11:26,055 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 07:11:26,055 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38568.72 MB 2025-02-14 07:11:26,055 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49058.68 MB 2025-02-14 07:11:26,055 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 07:11:26,056 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38223.91 MB 2025-02-14 07:11:26,229 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 07:11:26,231 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:11:26,231 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 07:11:26,232 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:11:26,232 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 07:11:26,237 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 07:11:26,238 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:11:26,238 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 07:11:26,238 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 07:12:57,085 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:12:57,085 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 07:12:57,090 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 07:12:57,094 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:12:57,094 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1774, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 07:12:57,095 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:12:57,095 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1774, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 07:13:24,428 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 07:13:24,428 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 07:13:24,428 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.33 seconds 2025-02-14 07:13:24,428 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:13:24,428 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33666.36 MB 2025-02-14 07:13:24,428 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39945.23 MB 2025-02-14 07:13:24,428 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6278.87 MB 2025-02-14 07:13:24,428 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61643.69 MB 2025-02-14 07:13:24,428 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45776.63 MB 2025-02-14 07:13:24,428 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15867.05 MB 2025-02-14 07:13:24,428 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48800.84 MB 2025-02-14 07:13:24,536 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 07:13:24,536 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 07:13:24,536 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 07:13:24,536 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:13:24,536 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39945.23 MB 2025-02-14 07:13:24,536 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33336.46 MB 2025-02-14 07:13:24,536 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6608.77 MB 2025-02-14 07:13:24,536 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45776.63 MB 2025-02-14 07:13:24,536 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 66265.81 MB 2025-02-14 07:13:24,536 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 20489.18 MB 2025-02-14 07:13:24,536 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 57385.61 MB 2025-02-14 07:13:26,508 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 07:13:26,509 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 07:13:26,509 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.97 seconds 2025-02-14 07:13:26,509 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:13:26,509 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33336.46 MB 2025-02-14 07:13:26,509 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33867.30 MB 2025-02-14 07:13:26,509 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 07:13:26,509 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 66265.81 MB 2025-02-14 07:13:26,509 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40913.34 MB 2025-02-14 07:13:26,509 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25352.47 MB 2025-02-14 07:13:26,509 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37845.98 MB 2025-02-14 07:13:26,523 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 07:13:26,523 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 07:13:26,523 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:13:26,523 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:13:26,523 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33867.30 MB 2025-02-14 07:13:26,523 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35756.83 MB 2025-02-14 07:13:26,523 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 07:13:26,523 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40913.34 MB 2025-02-14 07:13:26,523 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40915.44 MB 2025-02-14 07:13:26,523 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-14 07:13:26,523 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37174.26 MB 2025-02-14 07:13:26,737 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 07:13:26,737 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 07:13:26,737 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 07:13:26,737 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:13:26,737 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35756.83 MB 2025-02-14 07:13:26,737 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37998.69 MB 2025-02-14 07:13:26,737 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 07:13:26,737 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40915.44 MB 2025-02-14 07:13:26,737 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46105.89 MB 2025-02-14 07:13:26,737 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 07:13:26,737 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43543.81 MB 2025-02-14 07:13:26,738 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 07:13:26,738 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 07:13:26,738 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 07:13:26,738 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:13:26,738 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33867.30 MB 2025-02-14 07:13:26,738 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37998.69 MB 2025-02-14 07:13:26,738 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 07:13:26,738 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40913.34 MB 2025-02-14 07:13:26,738 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46105.89 MB 2025-02-14 07:13:26,738 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5192.55 MB 2025-02-14 07:13:26,738 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43543.81 MB 2025-02-14 07:13:26,912 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 07:13:26,913 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 07:13:26,913 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 07:13:26,913 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:13:26,913 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39533.07 MB 2025-02-14 07:13:26,913 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40300.08 MB 2025-02-14 07:13:26,913 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 07:13:26,913 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46105.89 MB 2025-02-14 07:13:26,913 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46523.22 MB 2025-02-14 07:13:26,913 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 07:13:26,913 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41007.86 MB 2025-02-14 07:13:26,932 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 07:13:26,932 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 07:13:26,932 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:13:26,932 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:13:26,932 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40712.96 MB 2025-02-14 07:13:26,932 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40941.75 MB 2025-02-14 07:13:26,932 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.79 MB 2025-02-14 07:13:26,932 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46523.22 MB 2025-02-14 07:13:26,932 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46523.22 MB 2025-02-14 07:13:26,932 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:13:26,932 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41155.83 MB 2025-02-14 07:13:26,933 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 07:13:26,933 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 07:13:26,933 - resource_logging.py:150 - __exit__ - DEBUG - Time: 29.84 seconds 2025-02-14 07:13:26,933 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:13:26,933 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27485.60 MB 2025-02-14 07:13:26,933 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41142.38 MB 2025-02-14 07:13:26,933 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13656.79 MB 2025-02-14 07:13:26,933 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61643.69 MB 2025-02-14 07:13:26,933 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46523.22 MB 2025-02-14 07:13:26,933 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15120.47 MB 2025-02-14 07:13:26,933 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41155.83 MB 2025-02-14 07:13:27,203 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 07:13:27,203 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 07:13:27,203 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 07:13:27,203 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:13:27,203 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41142.38 MB 2025-02-14 07:13:27,203 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32483.51 MB 2025-02-14 07:13:27,203 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8658.87 MB 2025-02-14 07:13:27,203 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46523.22 MB 2025-02-14 07:13:27,203 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46523.22 MB 2025-02-14 07:13:27,203 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:13:27,203 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43648.52 MB 2025-02-14 07:13:27,221 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8144, cut from 8146 2025-02-14 07:13:27,222 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 07:13:27,228 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 07:13:27,228 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 07:13:27,228 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:13:27,228 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:13:27,228 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32483.51 MB 2025-02-14 07:13:27,228 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40904.29 MB 2025-02-14 07:13:27,228 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8420.78 MB 2025-02-14 07:13:27,228 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46523.22 MB 2025-02-14 07:13:27,228 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54895.05 MB 2025-02-14 07:13:27,228 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8371.83 MB 2025-02-14 07:13:27,228 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40904.29 MB 2025-02-14 07:13:27,397 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7936] 2025-02-14 07:13:27,398 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:13:27,398 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 07:13:27,399 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:13:27,399 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 07:13:27,404 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 07:13:27,406 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:13:27,406 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 07:13:27,406 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 07:14:25,541 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:14:25,541 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 07:14:25,546 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 07:14:25,549 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:14:25,549 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2686, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 07:14:25,550 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:14:25,550 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2686, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 07:15:07,260 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 07:15:07,260 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 07:15:07,260 - resource_logging.py:150 - __exit__ - DEBUG - Time: 41.70 seconds 2025-02-14 07:15:07,260 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:15:07,260 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40021.32 MB 2025-02-14 07:15:07,260 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 49527.71 MB 2025-02-14 07:15:07,260 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 9506.39 MB 2025-02-14 07:15:07,260 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 81990.25 MB 2025-02-14 07:15:07,260 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50981.77 MB 2025-02-14 07:15:07,260 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31008.49 MB 2025-02-14 07:15:07,260 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 59033.31 MB 2025-02-14 07:15:07,386 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 07:15:07,386 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 07:15:07,386 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 07:15:07,386 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:15:07,386 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 49527.71 MB 2025-02-14 07:15:07,386 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38078.71 MB 2025-02-14 07:15:07,386 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -11449.00 MB 2025-02-14 07:15:07,386 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50981.77 MB 2025-02-14 07:15:07,386 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 68866.28 MB 2025-02-14 07:15:07,386 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 17884.51 MB 2025-02-14 07:15:07,386 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 60854.50 MB 2025-02-14 07:15:09,370 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 07:15:09,370 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 07:15:09,371 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.98 seconds 2025-02-14 07:15:09,371 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:15:09,371 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38078.71 MB 2025-02-14 07:15:09,371 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38609.55 MB 2025-02-14 07:15:09,371 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 07:15:09,371 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 68866.28 MB 2025-02-14 07:15:09,371 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40087.06 MB 2025-02-14 07:15:09,371 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28779.22 MB 2025-02-14 07:15:09,371 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42590.18 MB 2025-02-14 07:15:09,387 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 07:15:09,388 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 07:15:09,388 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:15:09,388 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:15:09,388 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38609.55 MB 2025-02-14 07:15:09,388 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40498.82 MB 2025-02-14 07:15:09,388 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.27 MB 2025-02-14 07:15:09,388 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40087.06 MB 2025-02-14 07:15:09,388 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43390.07 MB 2025-02-14 07:15:09,388 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 07:15:09,388 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41916.25 MB 2025-02-14 07:15:09,598 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 07:15:09,598 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 07:15:09,598 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 07:15:09,598 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:15:09,598 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40498.82 MB 2025-02-14 07:15:09,598 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42740.68 MB 2025-02-14 07:15:09,598 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 07:15:09,598 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43390.07 MB 2025-02-14 07:15:09,598 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50467.96 MB 2025-02-14 07:15:09,598 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7077.89 MB 2025-02-14 07:15:09,598 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48284.96 MB 2025-02-14 07:15:09,599 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 07:15:09,599 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 07:15:09,599 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 07:15:09,599 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:15:09,599 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38609.55 MB 2025-02-14 07:15:09,599 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42740.68 MB 2025-02-14 07:15:09,599 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.13 MB 2025-02-14 07:15:09,599 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40087.06 MB 2025-02-14 07:15:09,599 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50467.96 MB 2025-02-14 07:15:09,599 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10380.90 MB 2025-02-14 07:15:09,599 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48284.96 MB 2025-02-14 07:15:09,769 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 07:15:09,769 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 07:15:09,769 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 07:15:09,769 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:15:09,769 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44274.22 MB 2025-02-14 07:15:09,769 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45041.22 MB 2025-02-14 07:15:09,769 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 07:15:09,769 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50467.96 MB 2025-02-14 07:15:09,769 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50885.30 MB 2025-02-14 07:15:09,769 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 07:15:09,769 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45749.01 MB 2025-02-14 07:15:09,788 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 07:15:09,789 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 07:15:09,789 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:15:09,789 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:15:09,789 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45454.11 MB 2025-02-14 07:15:09,789 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45680.04 MB 2025-02-14 07:15:09,789 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 225.93 MB 2025-02-14 07:15:09,789 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50885.30 MB 2025-02-14 07:15:09,789 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50885.30 MB 2025-02-14 07:15:09,789 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:15:09,789 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45904.65 MB 2025-02-14 07:15:09,790 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 07:15:09,790 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 07:15:09,790 - resource_logging.py:150 - __exit__ - DEBUG - Time: 44.24 seconds 2025-02-14 07:15:09,790 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:15:09,790 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30663.08 MB 2025-02-14 07:15:09,790 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45881.12 MB 2025-02-14 07:15:09,790 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 15218.04 MB 2025-02-14 07:15:09,790 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 72628.57 MB 2025-02-14 07:15:09,790 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50885.30 MB 2025-02-14 07:15:09,790 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21743.27 MB 2025-02-14 07:15:09,790 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45904.65 MB 2025-02-14 07:15:10,062 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 07:15:10,063 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 07:15:10,063 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 07:15:10,063 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:15:10,063 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45881.12 MB 2025-02-14 07:15:10,063 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35667.01 MB 2025-02-14 07:15:10,063 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10214.11 MB 2025-02-14 07:15:10,063 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50885.30 MB 2025-02-14 07:15:10,063 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50885.30 MB 2025-02-14 07:15:10,063 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:15:10,063 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48392.78 MB 2025-02-14 07:15:10,081 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 07:15:10,081 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 07:15:10,087 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 07:15:10,087 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 07:15:10,087 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:15:10,087 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:15:10,087 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35667.01 MB 2025-02-14 07:15:10,087 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44105.70 MB 2025-02-14 07:15:10,088 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8438.69 MB 2025-02-14 07:15:10,088 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50885.30 MB 2025-02-14 07:15:10,088 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55081.70 MB 2025-02-14 07:15:10,088 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4196.40 MB 2025-02-14 07:15:10,088 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44105.70 MB 2025-02-14 07:15:10,250 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 07:15:10,252 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:15:10,252 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 07:15:10,253 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:15:10,253 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 07:15:10,258 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 07:15:10,259 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:15:10,259 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 07:15:10,259 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 07:16:12,174 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:16:12,174 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 07:16:12,179 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 07:16:12,182 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:16:12,182 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1506, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 07:16:12,183 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:16:12,183 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1506, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 07:16:35,521 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 07:16:35,521 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 07:16:35,521 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.33 seconds 2025-02-14 07:16:35,521 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:16:35,521 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31798.89 MB 2025-02-14 07:16:35,521 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37128.54 MB 2025-02-14 07:16:35,521 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5329.65 MB 2025-02-14 07:16:35,521 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63470.31 MB 2025-02-14 07:16:35,521 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45275.41 MB 2025-02-14 07:16:35,521 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18194.89 MB 2025-02-14 07:16:35,521 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46026.60 MB 2025-02-14 07:16:35,595 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 07:16:35,596 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 07:16:35,596 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 07:16:35,596 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:16:35,596 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37128.54 MB 2025-02-14 07:16:35,596 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31943.21 MB 2025-02-14 07:16:35,596 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5185.33 MB 2025-02-14 07:16:35,596 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45275.41 MB 2025-02-14 07:16:35,596 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50717.52 MB 2025-02-14 07:16:35,596 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5442.11 MB 2025-02-14 07:16:35,596 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46740.83 MB 2025-02-14 07:16:37,528 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 07:16:37,528 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 07:16:37,528 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 07:16:37,528 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:16:37,528 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31943.21 MB 2025-02-14 07:16:37,528 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32474.05 MB 2025-02-14 07:16:37,528 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 07:16:37,528 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50717.52 MB 2025-02-14 07:16:37,528 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37163.63 MB 2025-02-14 07:16:37,528 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13553.89 MB 2025-02-14 07:16:37,528 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36452.60 MB 2025-02-14 07:16:37,542 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 07:16:37,542 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 07:16:37,542 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:16:37,542 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:16:37,542 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32474.05 MB 2025-02-14 07:16:37,542 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34363.58 MB 2025-02-14 07:16:37,543 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 07:16:37,543 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37163.63 MB 2025-02-14 07:16:37,543 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38107.35 MB 2025-02-14 07:16:37,543 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 07:16:37,543 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35781.01 MB 2025-02-14 07:16:37,756 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 07:16:37,756 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 07:16:37,756 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 07:16:37,756 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:16:37,756 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34363.58 MB 2025-02-14 07:16:37,756 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36605.44 MB 2025-02-14 07:16:37,756 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 07:16:37,756 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38107.35 MB 2025-02-14 07:16:37,756 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44713.38 MB 2025-02-14 07:16:37,756 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 07:16:37,756 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42149.72 MB 2025-02-14 07:16:37,757 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 07:16:37,757 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 07:16:37,757 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 07:16:37,757 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:16:37,757 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32474.05 MB 2025-02-14 07:16:37,757 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36605.44 MB 2025-02-14 07:16:37,757 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 07:16:37,757 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37163.63 MB 2025-02-14 07:16:37,757 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44713.38 MB 2025-02-14 07:16:37,757 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-14 07:16:37,757 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42149.72 MB 2025-02-14 07:16:37,929 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 07:16:37,929 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 07:16:37,929 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 07:16:37,929 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:16:37,929 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38138.98 MB 2025-02-14 07:16:37,929 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38905.98 MB 2025-02-14 07:16:37,929 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 07:16:37,929 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44713.38 MB 2025-02-14 07:16:37,929 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45130.71 MB 2025-02-14 07:16:37,929 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 07:16:37,929 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39613.77 MB 2025-02-14 07:16:37,949 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 07:16:37,949 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 07:16:37,949 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:16:37,949 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:16:37,949 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39318.87 MB 2025-02-14 07:16:37,949 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39549.76 MB 2025-02-14 07:16:37,949 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 230.88 MB 2025-02-14 07:16:37,949 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45130.71 MB 2025-02-14 07:16:37,949 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45130.71 MB 2025-02-14 07:16:37,949 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:16:37,949 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39762.77 MB 2025-02-14 07:16:37,951 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 07:16:37,951 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 07:16:37,951 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.77 seconds 2025-02-14 07:16:37,951 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:16:37,951 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26551.86 MB 2025-02-14 07:16:37,951 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39750.83 MB 2025-02-14 07:16:37,951 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13198.96 MB 2025-02-14 07:16:37,951 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63470.31 MB 2025-02-14 07:16:37,951 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45130.71 MB 2025-02-14 07:16:37,951 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18339.59 MB 2025-02-14 07:16:37,951 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39762.77 MB 2025-02-14 07:16:38,223 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 07:16:38,223 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 07:16:38,223 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 07:16:38,223 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:16:38,223 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39750.83 MB 2025-02-14 07:16:38,223 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31555.91 MB 2025-02-14 07:16:38,223 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8194.92 MB 2025-02-14 07:16:38,223 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45130.71 MB 2025-02-14 07:16:38,223 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45130.71 MB 2025-02-14 07:16:38,223 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:16:38,223 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42262.61 MB 2025-02-14 07:16:38,241 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 07:16:38,242 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 07:16:38,248 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 07:16:38,248 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 07:16:38,248 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:16:38,248 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:16:38,248 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31555.91 MB 2025-02-14 07:16:38,248 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39994.60 MB 2025-02-14 07:16:38,248 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8438.69 MB 2025-02-14 07:16:38,248 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45130.71 MB 2025-02-14 07:16:38,248 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49327.11 MB 2025-02-14 07:16:38,248 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4196.40 MB 2025-02-14 07:16:38,248 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39994.60 MB 2025-02-14 07:16:38,411 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 07:16:38,413 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:16:38,413 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 07:16:38,414 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:16:38,414 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 07:16:38,419 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 07:16:38,420 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:16:38,420 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 07:16:38,420 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 07:17:27,765 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:17:27,765 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 07:17:27,770 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 07:17:27,774 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:17:27,774 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1344, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 07:17:27,775 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:17:27,775 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1344, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 07:17:48,636 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 07:17:48,636 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 07:17:48,636 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.85 seconds 2025-02-14 07:17:48,637 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:17:48,637 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30670.05 MB 2025-02-14 07:17:48,637 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35426.39 MB 2025-02-14 07:17:48,637 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4756.34 MB 2025-02-14 07:17:48,637 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61912.12 MB 2025-02-14 07:17:48,637 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44698.70 MB 2025-02-14 07:17:48,637 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17213.42 MB 2025-02-14 07:17:48,637 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44444.77 MB 2025-02-14 07:17:48,718 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 07:17:48,718 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 07:17:48,718 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 07:17:48,718 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:17:48,718 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35426.39 MB 2025-02-14 07:17:48,718 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31101.02 MB 2025-02-14 07:17:48,718 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4325.37 MB 2025-02-14 07:17:48,718 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44698.70 MB 2025-02-14 07:17:48,718 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54551.12 MB 2025-02-14 07:17:48,718 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9852.42 MB 2025-02-14 07:17:48,718 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49563.07 MB 2025-02-14 07:17:50,654 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 07:17:50,654 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 07:17:50,654 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 07:17:50,654 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:17:50,654 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31101.02 MB 2025-02-14 07:17:50,654 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31631.86 MB 2025-02-14 07:17:50,654 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 07:17:50,655 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54551.12 MB 2025-02-14 07:17:50,655 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39942.36 MB 2025-02-14 07:17:50,655 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14608.76 MB 2025-02-14 07:17:50,655 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35610.41 MB 2025-02-14 07:17:50,668 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 07:17:50,668 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 07:17:50,668 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:17:50,668 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:17:50,668 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31631.86 MB 2025-02-14 07:17:50,668 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33521.40 MB 2025-02-14 07:17:50,668 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 07:17:50,668 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39942.36 MB 2025-02-14 07:17:50,668 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39942.36 MB 2025-02-14 07:17:50,668 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:17:50,668 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34938.83 MB 2025-02-14 07:17:50,885 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 07:17:50,885 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 07:17:50,885 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 07:17:50,885 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:17:50,885 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33521.40 MB 2025-02-14 07:17:50,885 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35763.25 MB 2025-02-14 07:17:50,885 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 07:17:50,886 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39942.36 MB 2025-02-14 07:17:50,886 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43717.23 MB 2025-02-14 07:17:50,886 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 07:17:50,886 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41307.53 MB 2025-02-14 07:17:50,886 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 07:17:50,886 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 07:17:50,886 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 07:17:50,886 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:17:50,886 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31631.86 MB 2025-02-14 07:17:50,886 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35763.25 MB 2025-02-14 07:17:50,886 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 07:17:50,886 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39942.36 MB 2025-02-14 07:17:50,886 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43717.23 MB 2025-02-14 07:17:50,886 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 07:17:50,886 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41307.53 MB 2025-02-14 07:17:51,062 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 07:17:51,062 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 07:17:51,062 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 07:17:51,062 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:17:51,062 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37296.80 MB 2025-02-14 07:17:51,062 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38063.80 MB 2025-02-14 07:17:51,062 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 07:17:51,062 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43717.23 MB 2025-02-14 07:17:51,062 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44134.56 MB 2025-02-14 07:17:51,062 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 07:17:51,062 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38771.59 MB 2025-02-14 07:17:51,081 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 07:17:51,081 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 07:17:51,081 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:17:51,081 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:17:51,081 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38476.69 MB 2025-02-14 07:17:51,081 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38704.45 MB 2025-02-14 07:17:51,081 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.76 MB 2025-02-14 07:17:51,081 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44134.56 MB 2025-02-14 07:17:51,082 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44134.56 MB 2025-02-14 07:17:51,082 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:17:51,082 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38947.13 MB 2025-02-14 07:17:51,083 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 07:17:51,083 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 07:17:51,083 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.31 seconds 2025-02-14 07:17:51,083 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:17:51,083 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25987.44 MB 2025-02-14 07:17:51,083 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38904.81 MB 2025-02-14 07:17:51,083 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12917.36 MB 2025-02-14 07:17:51,083 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61912.12 MB 2025-02-14 07:17:51,083 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44134.56 MB 2025-02-14 07:17:51,083 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17777.56 MB 2025-02-14 07:17:51,083 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38947.13 MB 2025-02-14 07:17:51,351 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 07:17:51,351 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 07:17:51,351 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 07:17:51,351 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:17:51,351 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38904.81 MB 2025-02-14 07:17:51,351 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30980.32 MB 2025-02-14 07:17:51,351 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7924.48 MB 2025-02-14 07:17:51,351 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44134.56 MB 2025-02-14 07:17:51,351 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44134.56 MB 2025-02-14 07:17:51,351 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:17:51,351 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41407.56 MB 2025-02-14 07:17:51,368 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8133, cut from 8135 2025-02-14 07:17:51,369 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 07:17:51,375 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 07:17:51,375 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 07:17:51,375 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:17:51,375 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:17:51,375 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30980.32 MB 2025-02-14 07:17:51,375 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39389.61 MB 2025-02-14 07:17:51,375 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8409.29 MB 2025-02-14 07:17:51,375 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44134.56 MB 2025-02-14 07:17:51,375 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48314.19 MB 2025-02-14 07:17:51,375 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4179.62 MB 2025-02-14 07:17:51,375 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39389.61 MB 2025-02-14 07:17:51,536 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7925] 2025-02-14 07:17:51,538 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:17:51,538 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 07:17:51,539 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:17:51,539 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 07:17:51,543 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 07:17:51,544 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:17:51,544 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 07:17:51,545 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 07:18:01,567 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:18:01,568 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 07:18:01,573 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 07:18:01,576 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:18:01,576 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1146, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 07:18:01,577 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:18:01,577 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1146, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 07:18:19,551 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 07:18:19,551 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 07:18:19,551 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.97 seconds 2025-02-14 07:18:19,551 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:18:19,551 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29290.35 MB 2025-02-14 07:18:19,551 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33346.24 MB 2025-02-14 07:18:19,551 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4055.89 MB 2025-02-14 07:18:19,551 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56673.44 MB 2025-02-14 07:18:19,551 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35594.96 MB 2025-02-14 07:18:19,552 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21078.47 MB 2025-02-14 07:18:19,552 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42159.92 MB 2025-02-14 07:18:19,633 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 07:18:19,633 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 07:18:19,633 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 07:18:19,634 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:18:19,634 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33346.24 MB 2025-02-14 07:18:19,634 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30072.73 MB 2025-02-14 07:18:19,634 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3273.51 MB 2025-02-14 07:18:19,634 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35594.96 MB 2025-02-14 07:18:19,634 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54813.26 MB 2025-02-14 07:18:19,634 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 19218.30 MB 2025-02-14 07:18:19,634 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45599.67 MB 2025-02-14 07:18:21,609 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 07:18:21,609 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 07:18:21,609 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.97 seconds 2025-02-14 07:18:21,609 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:18:21,609 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30072.73 MB 2025-02-14 07:18:21,609 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30603.57 MB 2025-02-14 07:18:21,609 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 07:18:21,609 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54813.26 MB 2025-02-14 07:18:21,609 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34372.32 MB 2025-02-14 07:18:21,609 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20440.94 MB 2025-02-14 07:18:21,609 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34584.20 MB 2025-02-14 07:18:21,623 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 07:18:21,623 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 07:18:21,623 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:18:21,623 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:18:21,623 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30603.57 MB 2025-02-14 07:18:21,623 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32493.11 MB 2025-02-14 07:18:21,623 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 07:18:21,623 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34372.32 MB 2025-02-14 07:18:21,623 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36259.76 MB 2025-02-14 07:18:21,623 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 07:18:21,623 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33910.53 MB 2025-02-14 07:18:21,835 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 07:18:21,835 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 07:18:21,835 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 07:18:21,835 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:18:21,835 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32493.11 MB 2025-02-14 07:18:21,835 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34734.96 MB 2025-02-14 07:18:21,835 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 07:18:21,835 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36259.76 MB 2025-02-14 07:18:21,835 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42865.79 MB 2025-02-14 07:18:21,835 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 07:18:21,835 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40279.24 MB 2025-02-14 07:18:21,836 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 07:18:21,836 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 07:18:21,836 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 07:18:21,836 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:18:21,836 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30603.57 MB 2025-02-14 07:18:21,836 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34734.96 MB 2025-02-14 07:18:21,836 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 07:18:21,836 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34372.32 MB 2025-02-14 07:18:21,836 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42865.79 MB 2025-02-14 07:18:21,836 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8493.47 MB 2025-02-14 07:18:21,836 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40279.24 MB 2025-02-14 07:18:22,003 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 07:18:22,003 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 07:18:22,003 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 07:18:22,003 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:18:22,003 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36268.50 MB 2025-02-14 07:18:22,003 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37035.51 MB 2025-02-14 07:18:22,003 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 07:18:22,003 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42865.79 MB 2025-02-14 07:18:22,003 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43283.12 MB 2025-02-14 07:18:22,003 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 07:18:22,003 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37743.29 MB 2025-02-14 07:18:22,022 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 07:18:22,022 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 07:18:22,022 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:18:22,022 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:18:22,022 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37448.39 MB 2025-02-14 07:18:22,022 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37675.42 MB 2025-02-14 07:18:22,022 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.02 MB 2025-02-14 07:18:22,022 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43283.12 MB 2025-02-14 07:18:22,022 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43283.12 MB 2025-02-14 07:18:22,022 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:18:22,022 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37908.11 MB 2025-02-14 07:18:22,023 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 07:18:22,023 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 07:18:22,023 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.44 seconds 2025-02-14 07:18:22,023 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:18:22,023 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25297.60 MB 2025-02-14 07:18:22,023 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37876.02 MB 2025-02-14 07:18:22,023 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12578.43 MB 2025-02-14 07:18:22,023 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56673.44 MB 2025-02-14 07:18:22,023 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43283.12 MB 2025-02-14 07:18:22,023 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13390.32 MB 2025-02-14 07:18:22,023 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37908.11 MB 2025-02-14 07:18:22,295 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 07:18:22,295 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 07:18:22,295 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 07:18:22,295 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:18:22,295 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37876.02 MB 2025-02-14 07:18:22,295 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30295.28 MB 2025-02-14 07:18:22,295 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7580.75 MB 2025-02-14 07:18:22,295 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43283.12 MB 2025-02-14 07:18:22,295 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43283.12 MB 2025-02-14 07:18:22,295 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:18:22,295 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40382.84 MB 2025-02-14 07:18:22,313 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8143, cut from 8145 2025-02-14 07:18:22,313 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 07:18:22,319 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 07:18:22,319 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 07:18:22,320 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:18:22,320 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:18:22,320 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30295.28 MB 2025-02-14 07:18:22,320 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38714.35 MB 2025-02-14 07:18:22,320 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8419.08 MB 2025-02-14 07:18:22,320 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43283.12 MB 2025-02-14 07:18:22,320 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51654.95 MB 2025-02-14 07:18:22,320 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8371.83 MB 2025-02-14 07:18:22,320 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38714.35 MB 2025-02-14 07:18:22,481 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7935] 2025-02-14 07:18:22,483 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:18:22,483 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 07:18:22,484 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:18:22,484 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 07:18:22,489 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 07:18:22,490 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:18:22,490 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 07:18:22,490 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 07:18:31,562 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:18:31,563 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 07:18:31,568 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 07:18:31,571 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:18:31,571 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 196, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 07:18:31,572 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:18:31,572 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 196, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 07:18:34,691 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 07:18:34,691 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 07:18:34,691 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.12 seconds 2025-02-14 07:18:34,691 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:18:34,691 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22670.60 MB 2025-02-14 07:18:34,691 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23364.76 MB 2025-02-14 07:18:34,691 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 694.16 MB 2025-02-14 07:18:34,691 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60026.78 MB 2025-02-14 07:18:34,691 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25098.72 MB 2025-02-14 07:18:34,691 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34928.07 MB 2025-02-14 07:18:34,691 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32369.27 MB 2025-02-14 07:18:34,704 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 07:18:34,705 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 07:18:34,705 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:18:34,705 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:18:34,705 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23364.76 MB 2025-02-14 07:18:34,705 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23574.80 MB 2025-02-14 07:18:34,705 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 210.04 MB 2025-02-14 07:18:34,705 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25098.72 MB 2025-02-14 07:18:34,705 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27290.24 MB 2025-02-14 07:18:34,705 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2191.52 MB 2025-02-14 07:18:34,705 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25881.73 MB 2025-02-14 07:18:35,578 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 07:18:35,578 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 07:18:35,578 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.87 seconds 2025-02-14 07:18:35,578 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:18:35,578 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23574.80 MB 2025-02-14 07:18:35,578 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23811.02 MB 2025-02-14 07:18:35,578 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 236.22 MB 2025-02-14 07:18:35,578 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27290.24 MB 2025-02-14 07:18:35,578 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25656.56 MB 2025-02-14 07:18:35,578 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1633.68 MB 2025-02-14 07:18:35,578 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27746.53 MB 2025-02-14 07:18:35,586 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 07:18:35,586 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 07:18:35,586 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:18:35,586 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:18:35,586 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23811.02 MB 2025-02-14 07:18:35,586 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24651.66 MB 2025-02-14 07:18:35,586 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 840.64 MB 2025-02-14 07:18:35,586 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25656.56 MB 2025-02-14 07:18:35,586 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26499.61 MB 2025-02-14 07:18:35,586 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 843.06 MB 2025-02-14 07:18:35,586 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25282.42 MB 2025-02-14 07:18:35,683 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 07:18:35,683 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 07:18:35,683 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 07:18:35,683 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:18:35,683 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24651.66 MB 2025-02-14 07:18:35,683 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25649.33 MB 2025-02-14 07:18:35,683 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 997.66 MB 2025-02-14 07:18:35,683 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26499.61 MB 2025-02-14 07:18:35,683 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29452.40 MB 2025-02-14 07:18:35,683 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2952.79 MB 2025-02-14 07:18:35,683 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28116.50 MB 2025-02-14 07:18:35,684 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 07:18:35,684 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 07:18:35,684 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 07:18:35,684 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:18:35,684 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23811.02 MB 2025-02-14 07:18:35,684 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25649.33 MB 2025-02-14 07:18:35,684 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1838.30 MB 2025-02-14 07:18:35,684 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25656.56 MB 2025-02-14 07:18:35,684 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29452.40 MB 2025-02-14 07:18:35,684 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3795.85 MB 2025-02-14 07:18:35,684 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28116.50 MB 2025-02-14 07:18:35,759 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 07:18:35,759 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 07:18:35,759 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 07:18:35,759 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:18:35,759 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26331.75 MB 2025-02-14 07:18:35,759 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26673.07 MB 2025-02-14 07:18:35,759 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 341.32 MB 2025-02-14 07:18:35,759 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29452.40 MB 2025-02-14 07:18:35,759 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29636.95 MB 2025-02-14 07:18:35,759 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 184.55 MB 2025-02-14 07:18:35,759 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26995.48 MB 2025-02-14 07:18:35,769 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 07:18:35,769 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 07:18:35,769 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:18:35,769 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:18:35,770 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26856.81 MB 2025-02-14 07:18:35,770 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27061.07 MB 2025-02-14 07:18:35,770 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 204.26 MB 2025-02-14 07:18:35,770 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29636.95 MB 2025-02-14 07:18:35,770 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29641.15 MB 2025-02-14 07:18:35,770 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-14 07:18:35,770 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27096.13 MB 2025-02-14 07:18:35,771 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 07:18:35,771 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 07:18:35,771 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.20 seconds 2025-02-14 07:18:35,771 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:18:35,771 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21987.72 MB 2025-02-14 07:18:35,771 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27262.14 MB 2025-02-14 07:18:35,771 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5274.42 MB 2025-02-14 07:18:35,771 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60026.78 MB 2025-02-14 07:18:35,771 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29641.15 MB 2025-02-14 07:18:35,771 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30385.64 MB 2025-02-14 07:18:35,771 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27262.14 MB 2025-02-14 07:18:36,039 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 07:18:36,039 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 07:18:36,039 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 07:18:36,039 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:18:36,039 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27262.14 MB 2025-02-14 07:18:36,039 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25943.90 MB 2025-02-14 07:18:36,039 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1318.24 MB 2025-02-14 07:18:36,039 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29641.15 MB 2025-02-14 07:18:36,039 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29641.15 MB 2025-02-14 07:18:36,039 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:18:36,039 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27463.11 MB 2025-02-14 07:18:36,057 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 07:18:36,057 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 07:18:36,063 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 07:18:36,063 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 07:18:36,063 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:18:36,063 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:18:36,063 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25943.90 MB 2025-02-14 07:18:36,063 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34382.93 MB 2025-02-14 07:18:36,063 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 07:18:36,063 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29641.15 MB 2025-02-14 07:18:36,063 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40131.10 MB 2025-02-14 07:18:36,063 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 07:18:36,063 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34382.93 MB 2025-02-14 07:18:36,226 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 07:18:36,227 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:18:36,227 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 07:18:36,228 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:18:36,228 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 07:18:36,233 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 07:18:36,234 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:18:36,234 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 07:18:36,234 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 07:19:11,118 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:19:11,118 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 07:19:11,123 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 07:19:11,126 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:19:11,126 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 153, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 07:19:11,127 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:19:11,127 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 153, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 07:19:13,533 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 07:19:13,533 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 07:19:13,533 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.40 seconds 2025-02-14 07:19:13,533 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:19:13,533 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22370.97 MB 2025-02-14 07:19:13,533 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22912.43 MB 2025-02-14 07:19:13,533 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 541.46 MB 2025-02-14 07:19:13,533 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52716.11 MB 2025-02-14 07:19:13,533 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24758.98 MB 2025-02-14 07:19:13,533 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27957.13 MB 2025-02-14 07:19:13,533 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31843.15 MB 2025-02-14 07:19:13,545 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 07:19:13,545 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 07:19:13,545 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:19:13,545 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:19:13,545 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22912.43 MB 2025-02-14 07:19:13,545 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23120.18 MB 2025-02-14 07:19:13,545 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 207.75 MB 2025-02-14 07:19:13,545 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24758.98 MB 2025-02-14 07:19:13,545 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26543.65 MB 2025-02-14 07:19:13,545 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1784.68 MB 2025-02-14 07:19:13,545 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24984.22 MB 2025-02-14 07:19:14,256 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 07:19:14,256 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 07:19:14,256 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.71 seconds 2025-02-14 07:19:14,256 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:19:14,256 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23120.18 MB 2025-02-14 07:19:14,256 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23312.61 MB 2025-02-14 07:19:14,256 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 192.43 MB 2025-02-14 07:19:14,256 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26543.65 MB 2025-02-14 07:19:14,257 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24815.60 MB 2025-02-14 07:19:14,257 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1728.05 MB 2025-02-14 07:19:14,257 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27291.90 MB 2025-02-14 07:19:14,264 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 07:19:14,264 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 07:19:14,264 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 07:19:14,264 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:19:14,264 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23312.54 MB 2025-02-14 07:19:14,264 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23997.33 MB 2025-02-14 07:19:14,264 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 684.79 MB 2025-02-14 07:19:14,264 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24815.60 MB 2025-02-14 07:19:14,264 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25847.40 MB 2025-02-14 07:19:14,264 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1031.80 MB 2025-02-14 07:19:14,264 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24511.16 MB 2025-02-14 07:19:14,344 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 07:19:14,344 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 07:19:14,344 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 07:19:14,344 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:19:14,345 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23997.33 MB 2025-02-14 07:19:14,345 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24810.05 MB 2025-02-14 07:19:14,345 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 812.71 MB 2025-02-14 07:19:14,345 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25847.40 MB 2025-02-14 07:19:14,345 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28082.96 MB 2025-02-14 07:19:14,345 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2235.56 MB 2025-02-14 07:19:14,345 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26822.56 MB 2025-02-14 07:19:14,345 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 07:19:14,345 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 07:19:14,345 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 07:19:14,345 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:19:14,345 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23312.54 MB 2025-02-14 07:19:14,345 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24810.05 MB 2025-02-14 07:19:14,345 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1497.50 MB 2025-02-14 07:19:14,345 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24815.60 MB 2025-02-14 07:19:14,345 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28082.96 MB 2025-02-14 07:19:14,345 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3267.36 MB 2025-02-14 07:19:14,345 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26822.56 MB 2025-02-14 07:19:14,412 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 07:19:14,412 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 07:19:14,412 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 07:19:14,412 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:19:14,412 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25365.96 MB 2025-02-14 07:19:14,412 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25645.83 MB 2025-02-14 07:19:14,412 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 279.87 MB 2025-02-14 07:19:14,412 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28082.96 MB 2025-02-14 07:19:14,412 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28233.96 MB 2025-02-14 07:19:14,412 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 150.99 MB 2025-02-14 07:19:14,412 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25914.65 MB 2025-02-14 07:19:14,421 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 07:19:14,421 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 07:19:14,421 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:19:14,421 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:19:14,421 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25795.51 MB 2025-02-14 07:19:14,421 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26019.10 MB 2025-02-14 07:19:14,421 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 223.60 MB 2025-02-14 07:19:14,421 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28233.96 MB 2025-02-14 07:19:14,421 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28233.96 MB 2025-02-14 07:19:14,421 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:19:14,421 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26026.56 MB 2025-02-14 07:19:14,422 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 07:19:14,422 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 07:19:14,422 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.29 seconds 2025-02-14 07:19:14,422 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:19:14,422 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21837.90 MB 2025-02-14 07:19:14,422 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26220.03 MB 2025-02-14 07:19:14,422 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4382.12 MB 2025-02-14 07:19:14,422 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52716.11 MB 2025-02-14 07:19:14,422 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28233.96 MB 2025-02-14 07:19:14,422 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24482.15 MB 2025-02-14 07:19:14,422 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26220.03 MB 2025-02-14 07:19:14,690 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 07:19:14,690 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 07:19:14,690 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 07:19:14,690 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:19:14,690 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26220.03 MB 2025-02-14 07:19:14,690 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25637.96 MB 2025-02-14 07:19:14,690 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -582.07 MB 2025-02-14 07:19:14,690 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28233.96 MB 2025-02-14 07:19:14,690 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28368.18 MB 2025-02-14 07:19:14,690 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 134.22 MB 2025-02-14 07:19:14,690 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27324.71 MB 2025-02-14 07:19:14,708 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8156, cut from 8158 2025-02-14 07:19:14,708 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 07:19:14,714 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 07:19:14,715 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 07:19:14,715 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:19:14,715 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:19:14,715 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25637.96 MB 2025-02-14 07:19:14,715 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34071.26 MB 2025-02-14 07:19:14,715 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8433.30 MB 2025-02-14 07:19:14,715 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28368.18 MB 2025-02-14 07:19:14,715 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38849.74 MB 2025-02-14 07:19:14,715 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10481.57 MB 2025-02-14 07:19:14,715 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34071.26 MB 2025-02-14 07:19:14,878 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7948] 2025-02-14 07:19:14,879 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:19:14,879 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 07:19:14,880 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:19:14,880 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 07:19:14,885 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 07:19:14,886 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:19:14,886 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 07:19:14,886 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 07:20:07,354 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:20:07,354 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 07:20:07,359 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 07:20:07,363 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:20:07,363 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 770, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 07:20:07,364 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:20:07,364 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 770, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 07:20:19,300 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 07:20:19,300 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 07:20:19,300 - resource_logging.py:150 - __exit__ - DEBUG - Time: 11.93 seconds 2025-02-14 07:20:19,300 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:20:19,300 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26670.32 MB 2025-02-14 07:20:19,300 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29395.31 MB 2025-02-14 07:20:19,300 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2724.99 MB 2025-02-14 07:20:19,300 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47234.15 MB 2025-02-14 07:20:19,300 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31767.66 MB 2025-02-14 07:20:19,300 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15466.50 MB 2025-02-14 07:20:19,300 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38407.74 MB 2025-02-14 07:20:19,352 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 07:20:19,352 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 07:20:19,352 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-14 07:20:19,352 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:20:19,352 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29395.31 MB 2025-02-14 07:20:19,352 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28118.02 MB 2025-02-14 07:20:19,352 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1277.29 MB 2025-02-14 07:20:19,352 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31767.66 MB 2025-02-14 07:20:19,352 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43583.01 MB 2025-02-14 07:20:19,352 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 11815.35 MB 2025-02-14 07:20:19,352 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38419.04 MB 2025-02-14 07:20:21,287 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 07:20:21,287 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 07:20:21,287 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 07:20:21,287 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:20:21,287 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28118.02 MB 2025-02-14 07:20:21,288 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28648.86 MB 2025-02-14 07:20:21,288 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 07:20:21,288 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43583.01 MB 2025-02-14 07:20:21,288 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31165.78 MB 2025-02-14 07:20:21,288 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12417.24 MB 2025-02-14 07:20:21,288 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32628.45 MB 2025-02-14 07:20:21,302 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 07:20:21,302 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 07:20:21,302 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:20:21,302 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:20:21,302 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28648.86 MB 2025-02-14 07:20:21,302 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30538.40 MB 2025-02-14 07:20:21,302 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 07:20:21,302 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31165.78 MB 2025-02-14 07:20:21,302 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33996.93 MB 2025-02-14 07:20:21,302 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-14 07:20:21,302 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31955.83 MB 2025-02-14 07:20:21,514 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 07:20:21,514 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 07:20:21,514 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 07:20:21,514 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:20:21,514 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30538.40 MB 2025-02-14 07:20:21,514 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32780.25 MB 2025-02-14 07:20:21,514 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 07:20:21,514 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33996.93 MB 2025-02-14 07:20:21,514 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40602.96 MB 2025-02-14 07:20:21,515 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 07:20:21,515 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38324.54 MB 2025-02-14 07:20:21,515 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 07:20:21,515 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 07:20:21,515 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 07:20:21,515 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:20:21,515 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28648.86 MB 2025-02-14 07:20:21,515 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32780.25 MB 2025-02-14 07:20:21,515 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 07:20:21,515 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31165.78 MB 2025-02-14 07:20:21,515 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40602.96 MB 2025-02-14 07:20:21,515 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9437.18 MB 2025-02-14 07:20:21,515 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38324.54 MB 2025-02-14 07:20:21,683 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 07:20:21,683 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 07:20:21,683 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 07:20:21,683 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:20:21,683 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34313.80 MB 2025-02-14 07:20:21,683 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35080.80 MB 2025-02-14 07:20:21,683 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 07:20:21,683 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40602.96 MB 2025-02-14 07:20:21,683 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41020.29 MB 2025-02-14 07:20:21,683 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 07:20:21,683 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35788.59 MB 2025-02-14 07:20:21,702 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 07:20:21,702 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 07:20:21,702 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:20:21,702 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:20:21,702 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35493.69 MB 2025-02-14 07:20:21,702 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35720.81 MB 2025-02-14 07:20:21,702 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.12 MB 2025-02-14 07:20:21,702 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41020.29 MB 2025-02-14 07:20:21,702 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41020.29 MB 2025-02-14 07:20:21,702 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:20:21,702 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35909.28 MB 2025-02-14 07:20:21,703 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 07:20:21,703 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 07:20:21,703 - resource_logging.py:150 - __exit__ - DEBUG - Time: 14.34 seconds 2025-02-14 07:20:21,703 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:20:21,703 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23987.58 MB 2025-02-14 07:20:21,703 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35921.88 MB 2025-02-14 07:20:21,703 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11934.30 MB 2025-02-14 07:20:21,703 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47234.15 MB 2025-02-14 07:20:21,703 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41020.29 MB 2025-02-14 07:20:21,703 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6213.86 MB 2025-02-14 07:20:21,703 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35921.88 MB 2025-02-14 07:20:21,972 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 07:20:21,972 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 07:20:21,972 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 07:20:21,972 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:20:21,972 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35921.88 MB 2025-02-14 07:20:21,972 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28991.51 MB 2025-02-14 07:20:21,972 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6930.37 MB 2025-02-14 07:20:21,972 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41020.29 MB 2025-02-14 07:20:21,972 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41020.29 MB 2025-02-14 07:20:21,972 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:20:21,972 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38433.55 MB 2025-02-14 07:20:21,993 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 07:20:21,994 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 07:20:22,001 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 07:20:22,001 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 07:20:22,001 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 07:20:22,001 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:20:22,001 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28991.51 MB 2025-02-14 07:20:22,001 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37430.53 MB 2025-02-14 07:20:22,001 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 07:20:22,001 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41020.29 MB 2025-02-14 07:20:22,001 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51510.25 MB 2025-02-14 07:20:22,001 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 07:20:22,001 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37430.53 MB 2025-02-14 07:20:22,164 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 07:20:22,165 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:20:22,165 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 07:20:22,166 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:20:22,166 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 07:20:22,171 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 07:20:22,172 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:20:22,172 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 07:20:22,172 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 07:21:15,265 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:21:15,265 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 07:21:15,271 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 07:21:15,274 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:21:15,274 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1186, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 07:21:15,275 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:21:15,275 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1186, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 07:21:33,659 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 07:21:33,660 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 07:21:33,660 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.37 seconds 2025-02-14 07:21:33,660 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:21:33,660 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29569.73 MB 2025-02-14 07:21:33,660 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33766.92 MB 2025-02-14 07:21:33,660 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4197.19 MB 2025-02-14 07:21:33,660 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64095.26 MB 2025-02-14 07:21:33,660 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35341.21 MB 2025-02-14 07:21:33,660 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28754.05 MB 2025-02-14 07:21:33,660 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42665.79 MB 2025-02-14 07:21:33,749 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 07:21:33,749 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 07:21:33,749 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 07:21:33,749 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:21:33,749 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33766.92 MB 2025-02-14 07:21:33,749 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30281.33 MB 2025-02-14 07:21:33,749 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3485.59 MB 2025-02-14 07:21:33,749 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35341.21 MB 2025-02-14 07:21:33,749 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55792.63 MB 2025-02-14 07:21:33,749 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 20451.43 MB 2025-02-14 07:21:33,749 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46295.86 MB 2025-02-14 07:21:35,695 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 07:21:35,695 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 07:21:35,695 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-14 07:21:35,695 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:21:35,695 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30281.33 MB 2025-02-14 07:21:35,695 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30812.17 MB 2025-02-14 07:21:35,696 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 07:21:35,696 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55792.63 MB 2025-02-14 07:21:35,696 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33975.96 MB 2025-02-14 07:21:35,696 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21816.67 MB 2025-02-14 07:21:35,696 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34791.76 MB 2025-02-14 07:21:35,709 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 07:21:35,709 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 07:21:35,709 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:21:35,709 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:21:35,709 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30812.17 MB 2025-02-14 07:21:35,709 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32701.71 MB 2025-02-14 07:21:35,709 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 07:21:35,709 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33975.96 MB 2025-02-14 07:21:35,709 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35863.40 MB 2025-02-14 07:21:35,709 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 07:21:35,709 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34119.14 MB 2025-02-14 07:21:35,921 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 07:21:35,921 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 07:21:35,921 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 07:21:35,921 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:21:35,921 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32701.71 MB 2025-02-14 07:21:35,921 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34943.56 MB 2025-02-14 07:21:35,921 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 07:21:35,921 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35863.40 MB 2025-02-14 07:21:35,921 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42469.43 MB 2025-02-14 07:21:35,921 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 07:21:35,921 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40487.85 MB 2025-02-14 07:21:35,922 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 07:21:35,922 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 07:21:35,922 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 07:21:35,922 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:21:35,922 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30812.17 MB 2025-02-14 07:21:35,922 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34943.56 MB 2025-02-14 07:21:35,922 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 07:21:35,922 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33975.96 MB 2025-02-14 07:21:35,922 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42469.43 MB 2025-02-14 07:21:35,922 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8493.47 MB 2025-02-14 07:21:35,922 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40487.85 MB 2025-02-14 07:21:36,097 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 07:21:36,097 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 07:21:36,097 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 07:21:36,097 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:21:36,098 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36477.11 MB 2025-02-14 07:21:36,098 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37244.11 MB 2025-02-14 07:21:36,098 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 07:21:36,098 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42469.43 MB 2025-02-14 07:21:36,098 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42886.76 MB 2025-02-14 07:21:36,098 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 07:21:36,098 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37951.90 MB 2025-02-14 07:21:36,119 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 07:21:36,119 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 07:21:36,119 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:21:36,119 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:21:36,119 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37657.00 MB 2025-02-14 07:21:36,119 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37885.19 MB 2025-02-14 07:21:36,119 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.19 MB 2025-02-14 07:21:36,119 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42886.76 MB 2025-02-14 07:21:36,119 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42886.76 MB 2025-02-14 07:21:36,119 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:21:36,119 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38114.34 MB 2025-02-14 07:21:36,120 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 07:21:36,120 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 07:21:36,120 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.84 seconds 2025-02-14 07:21:36,120 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:21:36,120 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25437.61 MB 2025-02-14 07:21:36,120 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38085.47 MB 2025-02-14 07:21:36,120 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12647.86 MB 2025-02-14 07:21:36,120 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64095.26 MB 2025-02-14 07:21:36,120 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42886.76 MB 2025-02-14 07:21:36,120 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21208.50 MB 2025-02-14 07:21:36,120 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38114.34 MB 2025-02-14 07:21:36,388 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 07:21:36,388 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 07:21:36,388 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 07:21:36,388 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:21:36,388 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38085.47 MB 2025-02-14 07:21:36,388 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30429.74 MB 2025-02-14 07:21:36,388 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7655.73 MB 2025-02-14 07:21:36,388 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42886.76 MB 2025-02-14 07:21:36,389 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42886.76 MB 2025-02-14 07:21:36,389 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:21:36,389 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40587.31 MB 2025-02-14 07:21:36,407 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8130, cut from 8132 2025-02-14 07:21:36,407 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 07:21:36,413 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 07:21:36,413 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 07:21:36,413 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:21:36,413 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:21:36,413 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30429.74 MB 2025-02-14 07:21:36,413 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38835.40 MB 2025-02-14 07:21:36,413 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8405.66 MB 2025-02-14 07:21:36,413 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42886.76 MB 2025-02-14 07:21:36,413 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51246.01 MB 2025-02-14 07:21:36,413 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8359.25 MB 2025-02-14 07:21:36,413 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38835.40 MB 2025-02-14 07:21:36,577 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7922] 2025-02-14 07:21:36,579 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:21:36,579 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 07:21:36,580 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:21:36,580 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 07:21:36,585 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 07:21:36,586 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:21:36,586 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 07:21:36,586 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 07:22:34,597 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:22:34,597 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 07:22:34,602 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 07:22:34,606 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:22:34,606 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1221, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 07:22:34,607 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:22:34,607 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1221, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 07:22:53,462 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 07:22:53,462 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 07:22:53,462 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.85 seconds 2025-02-14 07:22:53,462 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:22:53,462 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29812.96 MB 2025-02-14 07:22:53,462 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34134.01 MB 2025-02-14 07:22:53,462 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4321.05 MB 2025-02-14 07:22:53,462 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59605.25 MB 2025-02-14 07:22:53,462 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39627.78 MB 2025-02-14 07:22:53,462 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19977.47 MB 2025-02-14 07:22:53,462 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43134.71 MB 2025-02-14 07:22:53,540 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 07:22:53,540 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 07:22:53,540 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 07:22:53,540 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:22:53,540 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34134.01 MB 2025-02-14 07:22:53,540 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30462.63 MB 2025-02-14 07:22:53,540 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3671.38 MB 2025-02-14 07:22:53,540 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39627.78 MB 2025-02-14 07:22:53,540 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53848.57 MB 2025-02-14 07:22:53,540 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 14220.79 MB 2025-02-14 07:22:53,540 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47006.72 MB 2025-02-14 07:22:55,491 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 07:22:55,491 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 07:22:55,491 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.95 seconds 2025-02-14 07:22:55,491 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:22:55,491 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30462.63 MB 2025-02-14 07:22:55,491 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30993.47 MB 2025-02-14 07:22:55,491 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 07:22:55,491 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53848.57 MB 2025-02-14 07:22:55,491 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33250.34 MB 2025-02-14 07:22:55,491 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20598.23 MB 2025-02-14 07:22:55,491 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34973.06 MB 2025-02-14 07:22:55,505 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 07:22:55,506 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 07:22:55,506 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:22:55,506 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:22:55,506 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30993.47 MB 2025-02-14 07:22:55,506 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32883.01 MB 2025-02-14 07:22:55,506 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 07:22:55,506 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33250.34 MB 2025-02-14 07:22:55,506 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36083.60 MB 2025-02-14 07:22:55,506 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2833.25 MB 2025-02-14 07:22:55,506 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34300.44 MB 2025-02-14 07:22:55,711 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 07:22:55,711 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 07:22:55,711 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 07:22:55,711 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:22:55,711 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32883.01 MB 2025-02-14 07:22:55,711 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35124.86 MB 2025-02-14 07:22:55,711 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 07:22:55,711 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36083.60 MB 2025-02-14 07:22:55,711 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42689.63 MB 2025-02-14 07:22:55,711 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 07:22:55,711 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40669.15 MB 2025-02-14 07:22:55,712 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 07:22:55,712 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 07:22:55,712 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 07:22:55,712 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:22:55,712 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30993.47 MB 2025-02-14 07:22:55,712 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35124.86 MB 2025-02-14 07:22:55,712 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 07:22:55,712 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33250.34 MB 2025-02-14 07:22:55,712 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42689.63 MB 2025-02-14 07:22:55,712 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9439.28 MB 2025-02-14 07:22:55,712 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40669.15 MB 2025-02-14 07:22:55,922 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 07:22:55,922 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 07:22:55,922 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 07:22:55,922 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:22:55,922 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36658.41 MB 2025-02-14 07:22:55,922 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37425.41 MB 2025-02-14 07:22:55,922 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 07:22:55,922 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42689.63 MB 2025-02-14 07:22:55,922 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43106.96 MB 2025-02-14 07:22:55,922 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 07:22:55,922 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38133.20 MB 2025-02-14 07:22:55,949 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 07:22:55,949 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 07:22:55,949 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:22:55,949 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:22:55,949 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37838.30 MB 2025-02-14 07:22:55,949 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38066.89 MB 2025-02-14 07:22:55,949 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.60 MB 2025-02-14 07:22:55,949 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43106.96 MB 2025-02-14 07:22:55,949 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43106.96 MB 2025-02-14 07:22:55,949 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:22:55,949 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38297.23 MB 2025-02-14 07:22:55,951 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 07:22:55,951 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 07:22:55,951 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.34 seconds 2025-02-14 07:22:55,951 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:22:55,951 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25558.90 MB 2025-02-14 07:22:55,951 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38267.87 MB 2025-02-14 07:22:55,951 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12708.96 MB 2025-02-14 07:22:55,951 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59605.25 MB 2025-02-14 07:22:55,951 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43106.96 MB 2025-02-14 07:22:55,951 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16498.29 MB 2025-02-14 07:22:55,951 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38297.23 MB 2025-02-14 07:22:56,236 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 07:22:56,236 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 07:22:56,236 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.28 seconds 2025-02-14 07:22:56,236 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:22:56,236 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38267.87 MB 2025-02-14 07:22:56,237 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30561.35 MB 2025-02-14 07:22:56,237 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7706.52 MB 2025-02-14 07:22:56,237 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43106.96 MB 2025-02-14 07:22:56,237 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43106.96 MB 2025-02-14 07:22:56,237 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:22:56,237 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40778.31 MB 2025-02-14 07:22:56,256 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8158, cut from 8160 2025-02-14 07:22:56,256 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 07:22:56,264 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 07:22:56,264 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 07:22:56,264 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:22:56,264 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:22:56,264 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30561.35 MB 2025-02-14 07:22:56,264 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38996.20 MB 2025-02-14 07:22:56,264 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8434.85 MB 2025-02-14 07:22:56,264 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43106.96 MB 2025-02-14 07:22:56,264 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51493.47 MB 2025-02-14 07:22:56,264 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8386.51 MB 2025-02-14 07:22:56,264 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38996.20 MB 2025-02-14 07:22:56,502 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7950] 2025-02-14 07:22:56,505 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:22:56,505 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 07:22:56,506 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:22:56,507 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 07:22:56,513 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 07:22:56,516 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:22:56,516 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 07:22:56,516 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 07:23:16,366 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:23:16,367 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 07:23:16,375 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 07:23:16,384 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:23:16,384 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1300, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 07:23:16,386 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:23:16,386 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1300, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 07:23:36,666 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 07:23:36,667 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 07:23:36,667 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.27 seconds 2025-02-14 07:23:36,667 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:23:36,667 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30363.45 MB 2025-02-14 07:23:36,667 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34964.60 MB 2025-02-14 07:23:36,667 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4601.15 MB 2025-02-14 07:23:36,667 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64072.19 MB 2025-02-14 07:23:36,667 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44107.30 MB 2025-02-14 07:23:36,667 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19964.89 MB 2025-02-14 07:23:36,667 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43911.68 MB 2025-02-14 07:23:36,745 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 07:23:36,745 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 07:23:36,745 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 07:23:36,745 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:23:36,745 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34964.60 MB 2025-02-14 07:23:36,745 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30872.28 MB 2025-02-14 07:23:36,745 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4092.32 MB 2025-02-14 07:23:36,745 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44107.30 MB 2025-02-14 07:23:36,745 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53257.18 MB 2025-02-14 07:23:36,745 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9149.87 MB 2025-02-14 07:23:36,745 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48521.30 MB 2025-02-14 07:23:38,671 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 07:23:38,671 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 07:23:38,671 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 07:23:38,671 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:23:38,671 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30872.28 MB 2025-02-14 07:23:38,671 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31403.12 MB 2025-02-14 07:23:38,671 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 07:23:38,671 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53257.18 MB 2025-02-14 07:23:38,671 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39506.15 MB 2025-02-14 07:23:38,671 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13751.03 MB 2025-02-14 07:23:38,671 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35381.67 MB 2025-02-14 07:23:38,699 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 07:23:38,699 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 07:23:38,699 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:23:38,699 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:23:38,699 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31403.12 MB 2025-02-14 07:23:38,699 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33292.65 MB 2025-02-14 07:23:38,699 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 07:23:38,699 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39506.15 MB 2025-02-14 07:23:38,699 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39508.25 MB 2025-02-14 07:23:38,699 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-14 07:23:38,699 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34710.08 MB 2025-02-14 07:23:38,908 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 07:23:38,908 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 07:23:38,908 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 07:23:38,908 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:23:38,908 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33292.65 MB 2025-02-14 07:23:38,908 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35534.51 MB 2025-02-14 07:23:38,908 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 07:23:38,908 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39508.25 MB 2025-02-14 07:23:38,908 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43754.98 MB 2025-02-14 07:23:38,908 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4246.73 MB 2025-02-14 07:23:38,908 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41078.79 MB 2025-02-14 07:23:38,909 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 07:23:38,909 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 07:23:38,909 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.24 seconds 2025-02-14 07:23:38,909 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:23:38,909 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31403.12 MB 2025-02-14 07:23:38,909 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35534.51 MB 2025-02-14 07:23:38,909 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 07:23:38,909 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39506.15 MB 2025-02-14 07:23:38,909 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43754.98 MB 2025-02-14 07:23:38,909 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4248.83 MB 2025-02-14 07:23:38,909 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41078.79 MB 2025-02-14 07:23:39,076 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 07:23:39,076 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 07:23:39,076 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 07:23:39,076 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:23:39,076 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37068.05 MB 2025-02-14 07:23:39,076 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37835.05 MB 2025-02-14 07:23:39,076 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 07:23:39,076 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43754.98 MB 2025-02-14 07:23:39,076 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44172.31 MB 2025-02-14 07:23:39,076 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 07:23:39,076 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38542.84 MB 2025-02-14 07:23:39,096 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 07:23:39,096 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 07:23:39,096 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:23:39,096 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:23:39,096 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38247.94 MB 2025-02-14 07:23:39,096 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38475.86 MB 2025-02-14 07:23:39,096 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.92 MB 2025-02-14 07:23:39,096 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44172.31 MB 2025-02-14 07:23:39,096 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44172.31 MB 2025-02-14 07:23:39,096 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:23:39,096 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38713.03 MB 2025-02-14 07:23:39,098 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 07:23:39,098 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 07:23:39,098 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.71 seconds 2025-02-14 07:23:39,098 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:23:39,098 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25834.14 MB 2025-02-14 07:23:39,098 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38675.75 MB 2025-02-14 07:23:39,098 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12841.61 MB 2025-02-14 07:23:39,098 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64072.19 MB 2025-02-14 07:23:39,098 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44172.31 MB 2025-02-14 07:23:39,098 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19899.88 MB 2025-02-14 07:23:39,098 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38713.03 MB 2025-02-14 07:23:39,366 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 07:23:39,366 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 07:23:39,366 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 07:23:39,366 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:23:39,366 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38675.75 MB 2025-02-14 07:23:39,366 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30820.37 MB 2025-02-14 07:23:39,366 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7855.38 MB 2025-02-14 07:23:39,367 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44172.31 MB 2025-02-14 07:23:39,367 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44172.31 MB 2025-02-14 07:23:39,367 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:23:39,367 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41172.67 MB 2025-02-14 07:23:39,384 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8114, cut from 8116 2025-02-14 07:23:39,385 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 07:23:39,391 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 07:23:39,391 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 07:23:39,391 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:23:39,391 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:23:39,391 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30820.37 MB 2025-02-14 07:23:39,391 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39209.52 MB 2025-02-14 07:23:39,391 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8389.15 MB 2025-02-14 07:23:39,391 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44172.31 MB 2025-02-14 07:23:39,391 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48343.55 MB 2025-02-14 07:23:39,391 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4171.24 MB 2025-02-14 07:23:39,391 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39209.52 MB 2025-02-14 07:23:39,544 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7906] 2025-02-14 07:23:39,545 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:23:39,545 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 07:23:39,546 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:23:39,546 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 07:23:39,550 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 07:23:39,551 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:23:39,552 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 07:23:39,552 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 07:24:19,425 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:24:19,425 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 07:24:19,430 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 07:24:19,433 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:24:19,433 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 363, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 07:24:19,434 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:24:19,434 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 363, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 07:24:25,051 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 07:24:25,052 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 07:24:25,052 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.61 seconds 2025-02-14 07:24:25,052 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:24:25,052 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23834.28 MB 2025-02-14 07:24:25,052 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25118.92 MB 2025-02-14 07:24:25,052 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1284.64 MB 2025-02-14 07:24:25,052 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56686.02 MB 2025-02-14 07:24:25,052 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29754.39 MB 2025-02-14 07:24:25,052 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26931.63 MB 2025-02-14 07:24:25,052 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33985.13 MB 2025-02-14 07:24:25,075 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 07:24:25,075 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 07:24:25,075 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:24:25,075 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:24:25,075 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25118.92 MB 2025-02-14 07:24:25,075 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25742.63 MB 2025-02-14 07:24:25,075 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 623.71 MB 2025-02-14 07:24:25,075 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29754.39 MB 2025-02-14 07:24:25,075 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33592.18 MB 2025-02-14 07:24:25,075 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3837.79 MB 2025-02-14 07:24:25,075 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30258.85 MB 2025-02-14 07:24:26,807 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 07:24:26,807 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 07:24:26,807 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.73 seconds 2025-02-14 07:24:26,807 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:24:26,807 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25742.63 MB 2025-02-14 07:24:26,807 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26224.36 MB 2025-02-14 07:24:26,807 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 481.74 MB 2025-02-14 07:24:26,807 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33592.18 MB 2025-02-14 07:24:26,807 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31362.91 MB 2025-02-14 07:24:26,807 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2229.27 MB 2025-02-14 07:24:26,807 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30167.08 MB 2025-02-14 07:24:26,820 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 07:24:26,820 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 07:24:26,820 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:24:26,820 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:24:26,820 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26224.36 MB 2025-02-14 07:24:26,820 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27938.72 MB 2025-02-14 07:24:26,820 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1714.36 MB 2025-02-14 07:24:26,820 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31362.91 MB 2025-02-14 07:24:26,820 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32220.64 MB 2025-02-14 07:24:26,820 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 857.74 MB 2025-02-14 07:24:26,820 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29225.04 MB 2025-02-14 07:24:27,007 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 07:24:27,007 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 07:24:27,007 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.19 seconds 2025-02-14 07:24:27,007 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:24:27,007 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27938.72 MB 2025-02-14 07:24:27,007 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29973.21 MB 2025-02-14 07:24:27,007 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2034.49 MB 2025-02-14 07:24:27,007 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32220.64 MB 2025-02-14 07:24:27,007 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37369.15 MB 2025-02-14 07:24:27,007 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5148.51 MB 2025-02-14 07:24:27,007 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35004.80 MB 2025-02-14 07:24:27,008 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 07:24:27,008 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 07:24:27,008 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 07:24:27,008 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:24:27,008 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26224.36 MB 2025-02-14 07:24:27,008 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29973.21 MB 2025-02-14 07:24:27,008 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3748.85 MB 2025-02-14 07:24:27,008 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31362.91 MB 2025-02-14 07:24:27,008 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37369.15 MB 2025-02-14 07:24:27,008 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6006.24 MB 2025-02-14 07:24:27,008 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35004.80 MB 2025-02-14 07:24:27,153 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 07:24:27,153 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 07:24:27,153 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-14 07:24:27,153 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:24:27,153 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31365.06 MB 2025-02-14 07:24:27,153 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32061.11 MB 2025-02-14 07:24:27,153 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 696.05 MB 2025-02-14 07:24:27,153 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37369.15 MB 2025-02-14 07:24:27,153 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37748.74 MB 2025-02-14 07:24:27,153 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 379.58 MB 2025-02-14 07:24:27,153 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32703.43 MB 2025-02-14 07:24:27,170 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 07:24:27,170 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 07:24:27,170 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:24:27,170 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:24:27,170 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32435.81 MB 2025-02-14 07:24:27,170 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32647.70 MB 2025-02-14 07:24:27,170 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 211.89 MB 2025-02-14 07:24:27,170 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37748.74 MB 2025-02-14 07:24:27,170 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37748.74 MB 2025-02-14 07:24:27,170 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:24:27,170 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32808.19 MB 2025-02-14 07:24:27,171 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 07:24:27,171 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 07:24:27,171 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.74 seconds 2025-02-14 07:24:27,171 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:24:27,171 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22569.56 MB 2025-02-14 07:24:27,171 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32848.77 MB 2025-02-14 07:24:27,171 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 10279.21 MB 2025-02-14 07:24:27,171 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56686.02 MB 2025-02-14 07:24:27,171 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37748.74 MB 2025-02-14 07:24:27,171 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18937.28 MB 2025-02-14 07:24:27,171 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32848.77 MB 2025-02-14 07:24:27,438 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 07:24:27,438 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 07:24:27,438 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 07:24:27,438 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:24:27,438 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32848.77 MB 2025-02-14 07:24:27,438 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27399.03 MB 2025-02-14 07:24:27,438 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5449.74 MB 2025-02-14 07:24:27,438 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37748.74 MB 2025-02-14 07:24:27,438 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37748.74 MB 2025-02-14 07:24:27,438 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:24:27,438 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35963.24 MB 2025-02-14 07:24:27,455 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 07:24:27,456 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 07:24:27,462 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 07:24:27,462 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 07:24:27,462 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:24:27,462 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:24:27,462 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27399.03 MB 2025-02-14 07:24:27,462 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35838.05 MB 2025-02-14 07:24:27,462 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 07:24:27,462 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37748.74 MB 2025-02-14 07:24:27,462 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46139.44 MB 2025-02-14 07:24:27,462 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 07:24:27,462 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35838.05 MB 2025-02-14 07:24:27,615 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 07:24:27,616 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:24:27,616 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 07:24:27,617 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:24:27,617 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 07:24:27,622 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 07:24:27,623 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:24:27,623 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 07:24:27,623 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 07:25:13,297 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:25:13,297 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 07:25:13,302 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 07:25:13,305 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:25:13,306 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 822, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 07:25:13,306 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:25:13,306 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 822, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 07:25:26,002 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 07:25:26,002 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 07:25:26,002 - resource_logging.py:150 - __exit__ - DEBUG - Time: 12.69 seconds 2025-02-14 07:25:26,002 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:25:26,002 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27032.67 MB 2025-02-14 07:25:26,002 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29941.68 MB 2025-02-14 07:25:26,002 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2909.01 MB 2025-02-14 07:25:26,002 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58724.45 MB 2025-02-14 07:25:26,002 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34028.39 MB 2025-02-14 07:25:26,002 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24696.06 MB 2025-02-14 07:25:26,002 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38769.77 MB 2025-02-14 07:25:26,056 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 07:25:26,056 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 07:25:26,056 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-14 07:25:26,056 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:25:26,056 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29941.68 MB 2025-02-14 07:25:26,056 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28388.36 MB 2025-02-14 07:25:26,056 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1553.33 MB 2025-02-14 07:25:26,056 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34028.39 MB 2025-02-14 07:25:26,056 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44820.33 MB 2025-02-14 07:25:26,056 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10791.94 MB 2025-02-14 07:25:26,056 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39695.87 MB 2025-02-14 07:25:27,968 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 07:25:27,968 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 07:25:27,968 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 07:25:27,968 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:25:27,968 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28388.36 MB 2025-02-14 07:25:27,968 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28919.20 MB 2025-02-14 07:25:27,968 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 07:25:27,968 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44820.33 MB 2025-02-14 07:25:27,968 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34586.23 MB 2025-02-14 07:25:27,968 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10234.10 MB 2025-02-14 07:25:27,968 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32897.74 MB 2025-02-14 07:25:27,982 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 07:25:27,982 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 07:25:27,982 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:25:27,982 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:25:27,982 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28919.20 MB 2025-02-14 07:25:27,982 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30808.73 MB 2025-02-14 07:25:27,982 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 07:25:27,982 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34586.23 MB 2025-02-14 07:25:27,982 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35529.95 MB 2025-02-14 07:25:27,982 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 07:25:27,982 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32226.16 MB 2025-02-14 07:25:28,186 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 07:25:28,186 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 07:25:28,186 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 07:25:28,186 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:25:28,186 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30808.73 MB 2025-02-14 07:25:28,186 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33050.59 MB 2025-02-14 07:25:28,186 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 07:25:28,186 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35529.95 MB 2025-02-14 07:25:28,186 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41192.26 MB 2025-02-14 07:25:28,186 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 07:25:28,186 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38594.87 MB 2025-02-14 07:25:28,187 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 07:25:28,187 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 07:25:28,187 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 07:25:28,187 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:25:28,187 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28919.20 MB 2025-02-14 07:25:28,187 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33050.59 MB 2025-02-14 07:25:28,187 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 07:25:28,187 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34586.23 MB 2025-02-14 07:25:28,187 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41192.26 MB 2025-02-14 07:25:28,187 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 07:25:28,187 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38594.87 MB 2025-02-14 07:25:28,352 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 07:25:28,352 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 07:25:28,352 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 07:25:28,352 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:25:28,352 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34584.13 MB 2025-02-14 07:25:28,352 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35351.13 MB 2025-02-14 07:25:28,352 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 07:25:28,352 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41192.26 MB 2025-02-14 07:25:28,352 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41609.59 MB 2025-02-14 07:25:28,352 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 07:25:28,352 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36058.92 MB 2025-02-14 07:25:28,371 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 07:25:28,371 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 07:25:28,371 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:25:28,371 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:25:28,371 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35764.02 MB 2025-02-14 07:25:28,371 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35991.74 MB 2025-02-14 07:25:28,371 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.72 MB 2025-02-14 07:25:28,371 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41609.59 MB 2025-02-14 07:25:28,371 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41609.59 MB 2025-02-14 07:25:28,371 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:25:28,371 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36185.72 MB 2025-02-14 07:25:28,372 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 07:25:28,372 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 07:25:28,373 - resource_logging.py:150 - __exit__ - DEBUG - Time: 15.06 seconds 2025-02-14 07:25:28,373 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:25:28,373 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24168.75 MB 2025-02-14 07:25:28,373 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36192.81 MB 2025-02-14 07:25:28,373 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12024.05 MB 2025-02-14 07:25:28,373 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58724.45 MB 2025-02-14 07:25:28,373 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41609.59 MB 2025-02-14 07:25:28,373 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17114.86 MB 2025-02-14 07:25:28,373 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36192.81 MB 2025-02-14 07:25:28,639 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 07:25:28,639 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 07:25:28,639 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 07:25:28,639 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:25:28,639 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36192.81 MB 2025-02-14 07:25:28,639 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29172.68 MB 2025-02-14 07:25:28,639 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7020.13 MB 2025-02-14 07:25:28,639 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41609.59 MB 2025-02-14 07:25:28,639 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41609.59 MB 2025-02-14 07:25:28,639 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:25:28,639 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38704.48 MB 2025-02-14 07:25:28,657 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 07:25:28,657 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 07:25:28,663 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 07:25:28,663 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 07:25:28,663 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:25:28,663 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:25:28,663 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29172.68 MB 2025-02-14 07:25:28,663 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37611.70 MB 2025-02-14 07:25:28,663 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 07:25:28,663 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41609.59 MB 2025-02-14 07:25:28,663 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50000.30 MB 2025-02-14 07:25:28,663 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 07:25:28,663 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37611.70 MB 2025-02-14 07:25:28,815 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 07:25:28,816 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:25:28,816 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 07:25:28,817 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:25:28,817 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 07:25:28,821 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 07:25:28,822 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:25:28,822 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 07:25:28,822 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 07:26:51,431 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:26:51,431 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 07:26:51,436 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 07:26:51,440 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:26:51,440 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1051, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 07:26:51,441 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:26:51,441 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1051, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 07:27:07,553 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 07:27:07,553 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 07:27:07,553 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.11 seconds 2025-02-14 07:27:07,553 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:27:07,553 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28628.38 MB 2025-02-14 07:27:07,553 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32348.72 MB 2025-02-14 07:27:07,553 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3720.35 MB 2025-02-14 07:27:07,553 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62585.31 MB 2025-02-14 07:27:07,553 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39030.10 MB 2025-02-14 07:27:07,554 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23555.21 MB 2025-02-14 07:27:07,554 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41271.45 MB 2025-02-14 07:27:07,619 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 07:27:07,619 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 07:27:07,619 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 07:27:07,619 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:27:07,619 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32348.72 MB 2025-02-14 07:27:07,619 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29577.81 MB 2025-02-14 07:27:07,619 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2770.92 MB 2025-02-14 07:27:07,619 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39030.10 MB 2025-02-14 07:27:07,619 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48934.94 MB 2025-02-14 07:27:07,619 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9904.85 MB 2025-02-14 07:27:07,619 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43995.01 MB 2025-02-14 07:27:09,536 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 07:27:09,536 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 07:27:09,536 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 07:27:09,536 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:27:09,536 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29577.81 MB 2025-02-14 07:27:09,536 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30108.65 MB 2025-02-14 07:27:09,536 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 07:27:09,536 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48934.94 MB 2025-02-14 07:27:09,536 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36725.33 MB 2025-02-14 07:27:09,536 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12209.62 MB 2025-02-14 07:27:09,536 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34087.19 MB 2025-02-14 07:27:09,550 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 07:27:09,550 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 07:27:09,550 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:27:09,550 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:27:09,550 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30108.65 MB 2025-02-14 07:27:09,550 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31998.18 MB 2025-02-14 07:27:09,550 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 07:27:09,550 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36725.33 MB 2025-02-14 07:27:09,550 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36727.42 MB 2025-02-14 07:27:09,550 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-14 07:27:09,550 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33415.61 MB 2025-02-14 07:27:09,757 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 07:27:09,757 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 07:27:09,757 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 07:27:09,757 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:27:09,757 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31998.18 MB 2025-02-14 07:27:09,757 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34240.04 MB 2025-02-14 07:27:09,757 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 07:27:09,757 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36727.42 MB 2025-02-14 07:27:09,757 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41917.87 MB 2025-02-14 07:27:09,757 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 07:27:09,757 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39784.32 MB 2025-02-14 07:27:09,758 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 07:27:09,758 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 07:27:09,758 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 07:27:09,758 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:27:09,758 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30108.65 MB 2025-02-14 07:27:09,758 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34240.04 MB 2025-02-14 07:27:09,758 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 07:27:09,758 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36725.33 MB 2025-02-14 07:27:09,758 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41917.87 MB 2025-02-14 07:27:09,758 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5192.55 MB 2025-02-14 07:27:09,758 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39784.32 MB 2025-02-14 07:27:09,924 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 07:27:09,924 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 07:27:09,924 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 07:27:09,924 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:27:09,924 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35773.58 MB 2025-02-14 07:27:09,924 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36540.58 MB 2025-02-14 07:27:09,924 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 07:27:09,924 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41917.87 MB 2025-02-14 07:27:09,924 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42335.21 MB 2025-02-14 07:27:09,924 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 07:27:09,924 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37248.37 MB 2025-02-14 07:27:09,943 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 07:27:09,943 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 07:27:09,943 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:27:09,943 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:27:09,943 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36953.47 MB 2025-02-14 07:27:09,943 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37181.38 MB 2025-02-14 07:27:09,943 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.91 MB 2025-02-14 07:27:09,943 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42335.21 MB 2025-02-14 07:27:09,943 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42335.21 MB 2025-02-14 07:27:09,943 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:27:09,943 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37415.36 MB 2025-02-14 07:27:09,944 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 07:27:09,944 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 07:27:09,944 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.50 seconds 2025-02-14 07:27:09,944 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:27:09,944 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24966.61 MB 2025-02-14 07:27:09,944 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37381.54 MB 2025-02-14 07:27:09,944 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12414.93 MB 2025-02-14 07:27:09,944 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62585.31 MB 2025-02-14 07:27:09,944 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42335.21 MB 2025-02-14 07:27:09,944 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20250.10 MB 2025-02-14 07:27:09,944 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37415.36 MB 2025-02-14 07:27:10,210 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 07:27:10,210 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 07:27:10,211 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 07:27:10,211 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:27:10,211 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37381.54 MB 2025-02-14 07:27:10,211 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29956.44 MB 2025-02-14 07:27:10,211 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7425.10 MB 2025-02-14 07:27:10,211 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42335.21 MB 2025-02-14 07:27:10,211 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42335.21 MB 2025-02-14 07:27:10,211 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:27:10,211 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39881.84 MB 2025-02-14 07:27:10,229 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8125, cut from 8127 2025-02-14 07:27:10,229 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 07:27:10,235 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 07:27:10,235 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 07:27:10,235 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:27:10,235 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:27:10,235 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29956.44 MB 2025-02-14 07:27:10,235 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38357.38 MB 2025-02-14 07:27:10,235 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8400.94 MB 2025-02-14 07:27:10,235 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42335.21 MB 2025-02-14 07:27:10,235 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50688.16 MB 2025-02-14 07:27:10,235 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8352.96 MB 2025-02-14 07:27:10,235 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38357.38 MB 2025-02-14 07:27:10,388 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7917] 2025-02-14 07:27:10,389 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:27:10,389 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 07:27:10,390 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:27:10,390 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 07:27:10,395 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 07:27:10,396 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:27:10,396 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 07:27:10,396 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 07:28:00,403 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:28:00,403 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 07:28:00,408 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 07:28:00,412 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:28:00,412 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1754, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 07:28:00,413 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:28:00,413 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1754, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 07:28:27,501 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 07:28:27,501 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 07:28:27,501 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.08 seconds 2025-02-14 07:28:27,501 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:28:27,501 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33526.99 MB 2025-02-14 07:28:27,501 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39734.56 MB 2025-02-14 07:28:27,501 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6207.57 MB 2025-02-14 07:28:27,501 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63216.55 MB 2025-02-14 07:28:27,501 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45655.00 MB 2025-02-14 07:28:27,501 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17561.55 MB 2025-02-14 07:28:27,501 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48661.48 MB 2025-02-14 07:28:27,608 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 07:28:27,608 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 07:28:27,608 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 07:28:27,608 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:28:27,608 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39734.56 MB 2025-02-14 07:28:27,608 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33232.48 MB 2025-02-14 07:28:27,608 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6502.08 MB 2025-02-14 07:28:27,608 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45655.00 MB 2025-02-14 07:28:27,608 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 66624.42 MB 2025-02-14 07:28:27,608 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 20969.42 MB 2025-02-14 07:28:27,608 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 57581.09 MB 2025-02-14 07:28:29,564 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 07:28:29,564 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 07:28:29,564 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.95 seconds 2025-02-14 07:28:29,564 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:28:29,564 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33232.48 MB 2025-02-14 07:28:29,564 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33763.32 MB 2025-02-14 07:28:29,564 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 07:28:29,564 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 66624.42 MB 2025-02-14 07:28:29,564 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36687.58 MB 2025-02-14 07:28:29,564 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29936.84 MB 2025-02-14 07:28:29,564 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37742.91 MB 2025-02-14 07:28:29,578 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 07:28:29,578 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 07:28:29,579 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:28:29,579 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:28:29,579 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33763.32 MB 2025-02-14 07:28:29,579 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35652.86 MB 2025-02-14 07:28:29,579 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 07:28:29,579 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36687.58 MB 2025-02-14 07:28:29,579 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39520.83 MB 2025-02-14 07:28:29,579 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2833.25 MB 2025-02-14 07:28:29,579 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37070.29 MB 2025-02-14 07:28:29,798 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 07:28:29,798 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 07:28:29,798 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 07:28:29,798 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:28:29,798 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35652.86 MB 2025-02-14 07:28:29,798 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37894.71 MB 2025-02-14 07:28:29,798 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 07:28:29,798 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39520.83 MB 2025-02-14 07:28:29,798 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46126.86 MB 2025-02-14 07:28:29,798 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 07:28:29,798 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43439.00 MB 2025-02-14 07:28:29,799 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 07:28:29,799 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 07:28:29,799 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 07:28:29,799 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:28:29,799 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33763.32 MB 2025-02-14 07:28:29,799 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37894.71 MB 2025-02-14 07:28:29,799 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 07:28:29,799 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36687.58 MB 2025-02-14 07:28:29,799 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46126.86 MB 2025-02-14 07:28:29,799 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9439.28 MB 2025-02-14 07:28:29,799 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43439.00 MB 2025-02-14 07:28:29,974 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 07:28:29,974 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 07:28:29,974 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 07:28:29,974 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:28:29,974 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39428.26 MB 2025-02-14 07:28:29,974 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40195.26 MB 2025-02-14 07:28:29,974 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 07:28:29,974 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46126.86 MB 2025-02-14 07:28:29,974 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46544.19 MB 2025-02-14 07:28:29,974 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 07:28:29,974 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40903.05 MB 2025-02-14 07:28:29,994 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 07:28:29,994 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 07:28:29,994 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:28:29,994 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:28:29,994 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40608.15 MB 2025-02-14 07:28:29,994 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40837.26 MB 2025-02-14 07:28:29,994 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.11 MB 2025-02-14 07:28:29,994 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46544.19 MB 2025-02-14 07:28:29,994 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46544.19 MB 2025-02-14 07:28:29,994 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:28:29,994 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41049.27 MB 2025-02-14 07:28:29,995 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 07:28:29,995 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 07:28:29,995 - resource_logging.py:150 - __exit__ - DEBUG - Time: 29.58 seconds 2025-02-14 07:28:29,995 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:28:29,995 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27415.92 MB 2025-02-14 07:28:29,995 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41038.18 MB 2025-02-14 07:28:29,995 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13622.26 MB 2025-02-14 07:28:29,995 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63216.55 MB 2025-02-14 07:28:29,995 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46544.19 MB 2025-02-14 07:28:29,995 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16672.36 MB 2025-02-14 07:28:29,995 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41049.27 MB 2025-02-14 07:28:30,265 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 07:28:30,265 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 07:28:30,265 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 07:28:30,265 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:28:30,265 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41038.18 MB 2025-02-14 07:28:30,266 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32417.56 MB 2025-02-14 07:28:30,266 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8620.62 MB 2025-02-14 07:28:30,266 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46544.19 MB 2025-02-14 07:28:30,266 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46544.19 MB 2025-02-14 07:28:30,266 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:28:30,266 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43548.01 MB 2025-02-14 07:28:30,284 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8156, cut from 8158 2025-02-14 07:28:30,284 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 07:28:30,290 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 07:28:30,290 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 07:28:30,290 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:28:30,290 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:28:30,290 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32417.56 MB 2025-02-14 07:28:30,290 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40850.86 MB 2025-02-14 07:28:30,290 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8433.30 MB 2025-02-14 07:28:30,290 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46544.19 MB 2025-02-14 07:28:30,290 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54928.61 MB 2025-02-14 07:28:30,290 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8384.41 MB 2025-02-14 07:28:30,290 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40850.86 MB 2025-02-14 07:28:30,462 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7948] 2025-02-14 07:28:30,463 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:28:30,463 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 07:28:30,464 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:28:30,464 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 07:28:30,469 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 07:28:30,470 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:28:30,470 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 07:28:30,470 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 07:28:38,926 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:28:38,926 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 07:28:38,931 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 07:28:38,934 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:28:38,935 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1195, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 07:28:38,935 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:28:38,935 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1195, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 07:28:57,638 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 07:28:57,638 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 07:28:57,638 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.70 seconds 2025-02-14 07:28:57,638 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:28:57,638 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29631.79 MB 2025-02-14 07:28:57,638 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33861.75 MB 2025-02-14 07:28:57,638 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4229.96 MB 2025-02-14 07:28:57,638 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63313.02 MB 2025-02-14 07:28:57,638 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35347.50 MB 2025-02-14 07:28:57,639 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27965.52 MB 2025-02-14 07:28:57,639 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42727.85 MB 2025-02-14 07:28:57,724 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 07:28:57,724 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 07:28:57,724 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 07:28:57,724 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:28:57,724 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33861.75 MB 2025-02-14 07:28:57,724 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30327.47 MB 2025-02-14 07:28:57,724 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3534.28 MB 2025-02-14 07:28:57,724 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35347.50 MB 2025-02-14 07:28:57,724 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56103.01 MB 2025-02-14 07:28:57,724 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 20755.51 MB 2025-02-14 07:28:57,724 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46524.10 MB 2025-02-14 07:28:59,685 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 07:28:59,685 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 07:28:59,685 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.96 seconds 2025-02-14 07:28:59,685 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:28:59,685 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30327.47 MB 2025-02-14 07:28:59,685 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30858.31 MB 2025-02-14 07:28:59,685 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 07:28:59,685 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56103.01 MB 2025-02-14 07:28:59,685 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33950.79 MB 2025-02-14 07:28:59,685 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22152.22 MB 2025-02-14 07:28:59,685 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34837.89 MB 2025-02-14 07:28:59,699 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 07:28:59,699 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 07:28:59,699 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:28:59,699 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:28:59,699 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30858.31 MB 2025-02-14 07:28:59,699 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32747.84 MB 2025-02-14 07:28:59,699 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 07:28:59,699 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33950.79 MB 2025-02-14 07:28:59,699 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35840.33 MB 2025-02-14 07:28:59,699 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1889.53 MB 2025-02-14 07:28:59,699 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34165.27 MB 2025-02-14 07:28:59,903 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 07:28:59,903 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 07:28:59,903 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 07:28:59,903 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:28:59,903 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32747.84 MB 2025-02-14 07:28:59,903 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34989.70 MB 2025-02-14 07:28:59,903 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 07:28:59,903 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35840.33 MB 2025-02-14 07:28:59,903 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42446.36 MB 2025-02-14 07:28:59,903 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 07:28:59,903 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40533.98 MB 2025-02-14 07:28:59,904 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 07:28:59,904 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 07:28:59,904 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 07:28:59,904 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:28:59,904 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30858.31 MB 2025-02-14 07:28:59,904 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34989.70 MB 2025-02-14 07:28:59,904 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 07:28:59,904 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33950.79 MB 2025-02-14 07:28:59,904 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42446.36 MB 2025-02-14 07:28:59,904 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8495.56 MB 2025-02-14 07:28:59,904 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40533.98 MB 2025-02-14 07:29:00,084 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 07:29:00,084 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 07:29:00,084 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 07:29:00,084 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:29:00,084 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36523.24 MB 2025-02-14 07:29:00,084 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37290.24 MB 2025-02-14 07:29:00,084 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 07:29:00,084 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42446.36 MB 2025-02-14 07:29:00,084 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42863.69 MB 2025-02-14 07:29:00,084 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 07:29:00,084 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37998.03 MB 2025-02-14 07:29:00,103 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 07:29:00,103 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 07:29:00,103 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:29:00,103 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:29:00,103 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37703.13 MB 2025-02-14 07:29:00,103 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37930.79 MB 2025-02-14 07:29:00,103 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.66 MB 2025-02-14 07:29:00,103 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42863.69 MB 2025-02-14 07:29:00,103 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42863.69 MB 2025-02-14 07:29:00,103 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:29:00,103 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38149.43 MB 2025-02-14 07:29:00,104 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 07:29:00,104 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 07:29:00,104 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.17 seconds 2025-02-14 07:29:00,104 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:29:00,104 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25468.32 MB 2025-02-14 07:29:00,104 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38130.83 MB 2025-02-14 07:29:00,104 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12662.52 MB 2025-02-14 07:29:00,104 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63313.02 MB 2025-02-14 07:29:00,104 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42863.69 MB 2025-02-14 07:29:00,104 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20449.33 MB 2025-02-14 07:29:00,104 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38149.43 MB 2025-02-14 07:29:00,378 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 07:29:00,378 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 07:29:00,378 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 07:29:00,378 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:29:00,378 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38130.83 MB 2025-02-14 07:29:00,378 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30456.75 MB 2025-02-14 07:29:00,378 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7674.08 MB 2025-02-14 07:29:00,378 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42863.69 MB 2025-02-14 07:29:00,378 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42863.69 MB 2025-02-14 07:29:00,378 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:29:00,378 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40629.60 MB 2025-02-14 07:29:00,396 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8120, cut from 8122 2025-02-14 07:29:00,397 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 07:29:00,403 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 07:29:00,403 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 07:29:00,403 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:29:00,403 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:29:00,403 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30456.75 MB 2025-02-14 07:29:00,403 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38853.40 MB 2025-02-14 07:29:00,403 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8396.64 MB 2025-02-14 07:29:00,403 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42863.69 MB 2025-02-14 07:29:00,403 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53297.02 MB 2025-02-14 07:29:00,403 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10433.33 MB 2025-02-14 07:29:00,403 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38853.40 MB 2025-02-14 07:29:00,553 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7912] 2025-02-14 07:29:00,554 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:29:00,554 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 07:29:00,555 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:29:00,555 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 07:29:00,560 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 07:29:00,561 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:29:00,561 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 07:29:00,561 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 07:29:41,155 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:29:41,156 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 07:29:41,160 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 07:29:41,164 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:29:41,164 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 186, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 07:29:41,166 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:29:41,166 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 186, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 07:29:44,062 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 07:29:44,062 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 07:29:44,062 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.89 seconds 2025-02-14 07:29:44,062 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:29:44,062 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22600.92 MB 2025-02-14 07:29:44,062 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23259.16 MB 2025-02-14 07:29:44,062 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 658.24 MB 2025-02-14 07:29:44,062 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61643.69 MB 2025-02-14 07:29:44,062 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25887.24 MB 2025-02-14 07:29:44,062 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35756.44 MB 2025-02-14 07:29:44,062 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32073.10 MB 2025-02-14 07:29:44,075 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 07:29:44,075 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 07:29:44,075 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:29:44,075 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:29:44,075 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23259.16 MB 2025-02-14 07:29:44,075 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23516.13 MB 2025-02-14 07:29:44,075 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 256.97 MB 2025-02-14 07:29:44,075 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25887.24 MB 2025-02-14 07:29:44,075 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27445.43 MB 2025-02-14 07:29:44,075 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1558.18 MB 2025-02-14 07:29:44,075 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25782.68 MB 2025-02-14 07:29:44,932 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 07:29:44,932 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 07:29:44,932 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.86 seconds 2025-02-14 07:29:44,932 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:29:44,932 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23516.13 MB 2025-02-14 07:29:44,932 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23751.03 MB 2025-02-14 07:29:44,932 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 234.90 MB 2025-02-14 07:29:44,932 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27445.43 MB 2025-02-14 07:29:44,932 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25641.88 MB 2025-02-14 07:29:44,932 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1803.55 MB 2025-02-14 07:29:44,932 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27687.86 MB 2025-02-14 07:29:44,941 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 07:29:44,941 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 07:29:44,941 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:29:44,941 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:29:44,941 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23750.96 MB 2025-02-14 07:29:44,941 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24586.88 MB 2025-02-14 07:29:44,941 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 835.92 MB 2025-02-14 07:29:44,941 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25641.88 MB 2025-02-14 07:29:44,941 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26482.84 MB 2025-02-14 07:29:44,941 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 840.96 MB 2025-02-14 07:29:44,941 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25214.09 MB 2025-02-14 07:29:45,034 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 07:29:45,034 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 07:29:45,034 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 07:29:45,034 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:29:45,034 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24586.88 MB 2025-02-14 07:29:45,034 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25578.93 MB 2025-02-14 07:29:45,034 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 992.06 MB 2025-02-14 07:29:45,034 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26482.84 MB 2025-02-14 07:29:45,034 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29418.85 MB 2025-02-14 07:29:45,034 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2936.01 MB 2025-02-14 07:29:45,034 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28034.08 MB 2025-02-14 07:29:45,035 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 07:29:45,035 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 07:29:45,035 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 07:29:45,035 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:29:45,035 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23750.96 MB 2025-02-14 07:29:45,035 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25578.93 MB 2025-02-14 07:29:45,035 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1827.97 MB 2025-02-14 07:29:45,035 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25641.88 MB 2025-02-14 07:29:45,035 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29418.85 MB 2025-02-14 07:29:45,035 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3776.97 MB 2025-02-14 07:29:45,035 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28034.08 MB 2025-02-14 07:29:45,107 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 07:29:45,107 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 07:29:45,107 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 07:29:45,107 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:29:45,107 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26257.53 MB 2025-02-14 07:29:45,107 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26597.84 MB 2025-02-14 07:29:45,107 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 340.32 MB 2025-02-14 07:29:45,107 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29418.85 MB 2025-02-14 07:29:45,107 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29603.40 MB 2025-02-14 07:29:45,107 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 184.55 MB 2025-02-14 07:29:45,107 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26918.24 MB 2025-02-14 07:29:45,116 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 07:29:45,117 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 07:29:45,117 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:29:45,117 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:29:45,117 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26780.55 MB 2025-02-14 07:29:45,117 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27004.24 MB 2025-02-14 07:29:45,117 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 223.69 MB 2025-02-14 07:29:45,117 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29603.40 MB 2025-02-14 07:29:45,117 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29603.40 MB 2025-02-14 07:29:45,117 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:29:45,117 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27034.93 MB 2025-02-14 07:29:45,118 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 07:29:45,118 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 07:29:45,118 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.95 seconds 2025-02-14 07:29:45,118 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:29:45,118 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21952.88 MB 2025-02-14 07:29:45,118 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27205.31 MB 2025-02-14 07:29:45,118 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5252.43 MB 2025-02-14 07:29:45,118 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61643.69 MB 2025-02-14 07:29:45,118 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29603.40 MB 2025-02-14 07:29:45,118 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32040.29 MB 2025-02-14 07:29:45,118 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27205.31 MB 2025-02-14 07:29:45,386 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 07:29:45,386 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 07:29:45,386 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 07:29:45,386 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:29:45,386 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27205.31 MB 2025-02-14 07:29:45,386 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25905.32 MB 2025-02-14 07:29:45,386 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1299.99 MB 2025-02-14 07:29:45,386 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29603.40 MB 2025-02-14 07:29:45,386 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29603.40 MB 2025-02-14 07:29:45,386 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:29:45,386 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27440.16 MB 2025-02-14 07:29:45,404 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 07:29:45,404 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 07:29:45,410 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 07:29:45,410 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 07:29:45,410 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:29:45,410 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:29:45,410 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25905.32 MB 2025-02-14 07:29:45,410 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34344.34 MB 2025-02-14 07:29:45,410 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 07:29:45,410 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29603.40 MB 2025-02-14 07:29:45,410 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40093.35 MB 2025-02-14 07:29:45,410 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 07:29:45,410 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34344.34 MB 2025-02-14 07:29:45,563 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 07:29:45,564 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:29:45,564 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 07:29:45,565 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:29:45,565 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 07:29:45,570 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 07:29:45,571 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:29:45,571 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 07:29:45,571 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 07:30:37,540 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:30:37,541 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 07:30:37,546 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 07:30:37,550 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:30:37,550 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1087, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 07:30:37,551 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:30:37,551 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1087, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 07:30:54,346 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 07:30:54,346 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 07:30:54,346 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.79 seconds 2025-02-14 07:30:54,346 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:30:54,346 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28879.23 MB 2025-02-14 07:30:54,346 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32726.06 MB 2025-02-14 07:30:54,346 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3846.83 MB 2025-02-14 07:30:54,346 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52678.36 MB 2025-02-14 07:30:54,346 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37086.04 MB 2025-02-14 07:30:54,346 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15592.33 MB 2025-02-14 07:30:54,346 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41604.22 MB 2025-02-14 07:30:54,414 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 07:30:54,414 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 07:30:54,414 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 07:30:54,414 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:30:54,414 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32726.06 MB 2025-02-14 07:30:54,414 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29764.96 MB 2025-02-14 07:30:54,414 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2961.10 MB 2025-02-14 07:30:54,414 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37086.04 MB 2025-02-14 07:30:54,414 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49016.73 MB 2025-02-14 07:30:54,414 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 11930.70 MB 2025-02-14 07:30:54,414 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43571.05 MB 2025-02-14 07:30:56,348 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 07:30:56,348 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 07:30:56,348 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 07:30:56,348 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:30:56,348 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29764.96 MB 2025-02-14 07:30:56,348 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30295.80 MB 2025-02-14 07:30:56,348 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 07:30:56,348 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49016.73 MB 2025-02-14 07:30:56,348 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34653.34 MB 2025-02-14 07:30:56,348 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14363.39 MB 2025-02-14 07:30:56,348 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34275.39 MB 2025-02-14 07:30:56,362 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 07:30:56,362 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 07:30:56,362 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:30:56,362 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:30:56,362 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30295.80 MB 2025-02-14 07:30:56,362 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32185.33 MB 2025-02-14 07:30:56,362 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 07:30:56,362 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34653.34 MB 2025-02-14 07:30:56,362 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36542.87 MB 2025-02-14 07:30:56,362 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1889.53 MB 2025-02-14 07:30:56,362 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33602.76 MB 2025-02-14 07:30:56,579 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 07:30:56,579 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 07:30:56,579 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 07:30:56,579 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:30:56,579 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32185.33 MB 2025-02-14 07:30:56,579 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34427.19 MB 2025-02-14 07:30:56,579 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 07:30:56,579 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36542.87 MB 2025-02-14 07:30:56,579 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42205.18 MB 2025-02-14 07:30:56,579 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 07:30:56,579 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39971.47 MB 2025-02-14 07:30:56,580 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 07:30:56,580 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 07:30:56,580 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 07:30:56,580 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:30:56,580 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30295.80 MB 2025-02-14 07:30:56,580 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34427.19 MB 2025-02-14 07:30:56,580 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 07:30:56,580 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34653.34 MB 2025-02-14 07:30:56,580 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42205.18 MB 2025-02-14 07:30:56,580 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7551.84 MB 2025-02-14 07:30:56,580 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39971.47 MB 2025-02-14 07:30:56,749 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 07:30:56,749 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 07:30:56,749 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 07:30:56,749 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:30:56,749 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35960.73 MB 2025-02-14 07:30:56,749 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36727.73 MB 2025-02-14 07:30:56,750 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 07:30:56,750 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42205.18 MB 2025-02-14 07:30:56,750 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42622.52 MB 2025-02-14 07:30:56,750 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 07:30:56,750 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37435.52 MB 2025-02-14 07:30:56,769 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 07:30:56,769 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 07:30:56,769 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:30:56,769 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:30:56,769 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37140.62 MB 2025-02-14 07:30:56,769 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37369.27 MB 2025-02-14 07:30:56,769 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.65 MB 2025-02-14 07:30:56,769 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42622.52 MB 2025-02-14 07:30:56,769 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42622.52 MB 2025-02-14 07:30:56,769 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:30:56,769 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37604.35 MB 2025-02-14 07:30:56,770 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 07:30:56,770 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 07:30:56,770 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.22 seconds 2025-02-14 07:30:56,770 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:30:56,770 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25092.04 MB 2025-02-14 07:30:56,770 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37570.29 MB 2025-02-14 07:30:56,770 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12478.26 MB 2025-02-14 07:30:56,770 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52678.36 MB 2025-02-14 07:30:56,770 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42622.52 MB 2025-02-14 07:30:56,770 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10055.84 MB 2025-02-14 07:30:56,770 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37604.35 MB 2025-02-14 07:30:57,040 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 07:30:57,040 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 07:30:57,040 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 07:30:57,040 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:30:57,040 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37570.29 MB 2025-02-14 07:30:57,040 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30095.20 MB 2025-02-14 07:30:57,040 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7475.09 MB 2025-02-14 07:30:57,040 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42622.52 MB 2025-02-14 07:30:57,040 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42622.52 MB 2025-02-14 07:30:57,040 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:30:57,040 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40081.35 MB 2025-02-14 07:30:57,058 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8160, cut from 8162 2025-02-14 07:30:57,058 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 07:30:57,064 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 07:30:57,065 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 07:30:57,065 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:30:57,065 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:30:57,065 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30095.20 MB 2025-02-14 07:30:57,065 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38532.67 MB 2025-02-14 07:30:57,065 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8437.47 MB 2025-02-14 07:30:57,065 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42622.52 MB 2025-02-14 07:30:57,065 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51011.13 MB 2025-02-14 07:30:57,065 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8388.61 MB 2025-02-14 07:30:57,065 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38532.67 MB 2025-02-14 07:30:57,227 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7952] 2025-02-14 07:30:57,229 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:30:57,229 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 07:30:57,230 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:30:57,230 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 07:30:57,235 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 07:30:57,236 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:30:57,236 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 07:30:57,236 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 07:31:57,783 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:31:57,784 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 07:31:57,789 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 07:31:57,794 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:31:57,794 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1587, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 07:31:57,795 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:31:57,795 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1587, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 07:32:22,376 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 07:32:22,377 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 07:32:22,377 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.57 seconds 2025-02-14 07:32:22,377 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:32:22,377 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32363.31 MB 2025-02-14 07:32:22,377 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37979.62 MB 2025-02-14 07:32:22,377 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5616.30 MB 2025-02-14 07:32:22,377 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59399.73 MB 2025-02-14 07:32:22,377 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45145.39 MB 2025-02-14 07:32:22,377 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14254.34 MB 2025-02-14 07:32:22,377 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46818.32 MB 2025-02-14 07:32:22,427 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 07:32:22,427 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 07:32:22,427 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-14 07:32:22,427 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:32:22,427 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37979.62 MB 2025-02-14 07:32:22,427 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30334.65 MB 2025-02-14 07:32:22,427 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7644.97 MB 2025-02-14 07:32:22,427 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45145.39 MB 2025-02-14 07:32:22,427 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45145.39 MB 2025-02-14 07:32:22,427 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:32:22,427 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39549.34 MB 2025-02-14 07:32:22,966 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 07:32:22,966 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 07:32:22,966 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.54 seconds 2025-02-14 07:32:22,966 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:32:22,966 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30334.65 MB 2025-02-14 07:32:22,967 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30481.96 MB 2025-02-14 07:32:22,967 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 147.31 MB 2025-02-14 07:32:22,967 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45145.39 MB 2025-02-14 07:32:22,967 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39527.12 MB 2025-02-14 07:32:22,967 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5618.27 MB 2025-02-14 07:32:22,967 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34419.36 MB 2025-02-14 07:32:22,973 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 07:32:22,973 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 07:32:22,973 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 07:32:22,973 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:32:22,973 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30481.96 MB 2025-02-14 07:32:22,973 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31006.18 MB 2025-02-14 07:32:22,973 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 524.22 MB 2025-02-14 07:32:22,973 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39527.12 MB 2025-02-14 07:32:22,973 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39529.22 MB 2025-02-14 07:32:22,973 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-14 07:32:22,973 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31400.51 MB 2025-02-14 07:32:23,083 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 07:32:23,083 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 07:32:23,083 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 07:32:23,083 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:32:23,083 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31006.18 MB 2025-02-14 07:32:23,083 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31644.52 MB 2025-02-14 07:32:23,083 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 638.35 MB 2025-02-14 07:32:23,083 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39529.22 MB 2025-02-14 07:32:23,083 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39529.22 MB 2025-02-14 07:32:23,083 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:32:23,083 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33167.82 MB 2025-02-14 07:32:23,084 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 07:32:23,084 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 07:32:23,084 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 07:32:23,084 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:32:23,084 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30481.96 MB 2025-02-14 07:32:23,084 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31644.52 MB 2025-02-14 07:32:23,084 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1162.57 MB 2025-02-14 07:32:23,084 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39527.12 MB 2025-02-14 07:32:23,084 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39529.22 MB 2025-02-14 07:32:23,084 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-14 07:32:23,084 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33167.82 MB 2025-02-14 07:32:23,141 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 07:32:23,141 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 07:32:23,141 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-14 07:32:23,141 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:32:23,141 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32259.48 MB 2025-02-14 07:32:23,141 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32526.88 MB 2025-02-14 07:32:23,141 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 267.40 MB 2025-02-14 07:32:23,141 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39529.22 MB 2025-02-14 07:32:23,141 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39701.18 MB 2025-02-14 07:32:23,141 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 171.97 MB 2025-02-14 07:32:23,141 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32723.29 MB 2025-02-14 07:32:23,148 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 07:32:23,148 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 07:32:23,148 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 07:32:23,148 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:32:23,148 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32696.03 MB 2025-02-14 07:32:23,148 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32924.58 MB 2025-02-14 07:32:23,148 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.55 MB 2025-02-14 07:32:23,148 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39701.18 MB 2025-02-14 07:32:23,148 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39701.18 MB 2025-02-14 07:32:23,148 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:32:23,148 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32924.58 MB 2025-02-14 07:32:23,149 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 07:32:23,149 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 07:32:23,149 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.35 seconds 2025-02-14 07:32:23,149 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:32:23,149 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26834.08 MB 2025-02-14 07:32:23,149 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33125.09 MB 2025-02-14 07:32:23,149 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6291.01 MB 2025-02-14 07:32:23,149 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59399.73 MB 2025-02-14 07:32:23,149 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39701.18 MB 2025-02-14 07:32:23,150 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19698.55 MB 2025-02-14 07:32:23,150 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33125.09 MB 2025-02-14 07:32:23,418 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 07:32:23,418 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 07:32:23,418 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 07:32:23,418 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:32:23,418 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33125.09 MB 2025-02-14 07:32:23,418 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36130.64 MB 2025-02-14 07:32:23,418 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3005.55 MB 2025-02-14 07:32:23,418 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39701.18 MB 2025-02-14 07:32:23,418 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39701.18 MB 2025-02-14 07:32:23,418 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:32:23,418 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36431.16 MB 2025-02-14 07:32:23,437 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8139, cut from 8141 2025-02-14 07:32:23,437 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 07:32:23,443 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 07:32:23,443 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 07:32:23,443 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:32:23,443 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:32:23,443 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30567.46 MB 2025-02-14 07:32:23,443 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38982.41 MB 2025-02-14 07:32:23,443 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8414.95 MB 2025-02-14 07:32:23,443 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39701.18 MB 2025-02-14 07:32:23,443 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48068.82 MB 2025-02-14 07:32:23,443 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8367.64 MB 2025-02-14 07:32:23,443 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38982.41 MB 2025-02-14 07:32:23,601 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7931] 2025-02-14 07:32:23,603 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:32:23,603 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 07:32:23,604 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:32:23,604 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 07:32:23,608 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 07:32:23,610 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:32:23,610 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 07:32:23,610 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 07:32:35,197 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:32:35,197 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 07:32:35,202 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 07:32:35,205 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:32:35,205 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1290, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 07:32:35,206 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:32:35,206 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1290, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 07:32:55,353 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 07:32:55,353 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 07:32:55,353 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.14 seconds 2025-02-14 07:32:55,353 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:32:55,353 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30293.77 MB 2025-02-14 07:32:55,353 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34859.27 MB 2025-02-14 07:32:55,353 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4565.50 MB 2025-02-14 07:32:55,353 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56436.46 MB 2025-02-14 07:32:55,353 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44061.16 MB 2025-02-14 07:32:55,353 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12375.29 MB 2025-02-14 07:32:55,353 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43842.00 MB 2025-02-14 07:32:55,430 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 07:32:55,430 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 07:32:55,430 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 07:32:55,430 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:32:55,430 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34859.27 MB 2025-02-14 07:32:55,430 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30820.29 MB 2025-02-14 07:32:55,430 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4038.97 MB 2025-02-14 07:32:55,430 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44061.16 MB 2025-02-14 07:32:55,430 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53299.12 MB 2025-02-14 07:32:55,430 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9237.95 MB 2025-02-14 07:32:55,430 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48478.16 MB 2025-02-14 07:32:57,367 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 07:32:57,367 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 07:32:57,367 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-14 07:32:57,367 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:32:57,367 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30820.29 MB 2025-02-14 07:32:57,367 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31351.13 MB 2025-02-14 07:32:57,367 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 07:32:57,367 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53299.12 MB 2025-02-14 07:32:57,367 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35311.85 MB 2025-02-14 07:32:57,367 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17987.27 MB 2025-02-14 07:32:57,367 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35330.72 MB 2025-02-14 07:32:57,382 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 07:32:57,382 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 07:32:57,382 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:32:57,382 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:32:57,382 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31351.13 MB 2025-02-14 07:32:57,382 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33240.67 MB 2025-02-14 07:32:57,382 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 07:32:57,382 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35311.85 MB 2025-02-14 07:32:57,382 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37201.38 MB 2025-02-14 07:32:57,382 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1889.53 MB 2025-02-14 07:32:57,382 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34658.10 MB 2025-02-14 07:32:57,596 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 07:32:57,596 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 07:32:57,596 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 07:32:57,596 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:32:57,596 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33240.67 MB 2025-02-14 07:32:57,596 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35482.52 MB 2025-02-14 07:32:57,596 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 07:32:57,596 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37201.38 MB 2025-02-14 07:32:57,596 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43807.41 MB 2025-02-14 07:32:57,596 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 07:32:57,596 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41026.81 MB 2025-02-14 07:32:57,597 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 07:32:57,597 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 07:32:57,597 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 07:32:57,597 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:32:57,597 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31351.13 MB 2025-02-14 07:32:57,597 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35482.52 MB 2025-02-14 07:32:57,597 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 07:32:57,597 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35311.85 MB 2025-02-14 07:32:57,597 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43807.41 MB 2025-02-14 07:32:57,597 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8495.56 MB 2025-02-14 07:32:57,597 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41026.81 MB 2025-02-14 07:32:57,768 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 07:32:57,768 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 07:32:57,768 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 07:32:57,768 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:32:57,768 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37016.07 MB 2025-02-14 07:32:57,768 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37783.07 MB 2025-02-14 07:32:57,768 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 07:32:57,768 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43807.41 MB 2025-02-14 07:32:57,768 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44224.74 MB 2025-02-14 07:32:57,768 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 07:32:57,768 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38490.86 MB 2025-02-14 07:32:57,788 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 07:32:57,788 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 07:32:57,788 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:32:57,788 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:32:57,788 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38195.96 MB 2025-02-14 07:32:57,788 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38423.55 MB 2025-02-14 07:32:57,788 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.59 MB 2025-02-14 07:32:57,788 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44224.74 MB 2025-02-14 07:32:57,788 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44224.74 MB 2025-02-14 07:32:57,788 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:32:57,788 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38668.18 MB 2025-02-14 07:32:57,789 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 07:32:57,789 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 07:32:57,789 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.58 seconds 2025-02-14 07:32:57,789 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:32:57,790 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25799.30 MB 2025-02-14 07:32:57,790 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38623.51 MB 2025-02-14 07:32:57,790 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12824.21 MB 2025-02-14 07:32:57,790 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56436.46 MB 2025-02-14 07:32:57,790 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44224.74 MB 2025-02-14 07:32:57,790 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12211.72 MB 2025-02-14 07:32:57,790 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38668.18 MB 2025-02-14 07:32:58,061 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 07:32:58,061 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 07:32:58,061 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 07:32:58,061 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:32:58,061 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38623.51 MB 2025-02-14 07:32:58,061 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30787.45 MB 2025-02-14 07:32:58,061 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7836.07 MB 2025-02-14 07:32:58,061 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44224.74 MB 2025-02-14 07:32:58,061 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44224.74 MB 2025-02-14 07:32:58,061 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:32:58,061 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41121.36 MB 2025-02-14 07:32:58,079 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8117, cut from 8119 2025-02-14 07:32:58,079 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 07:32:58,085 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 07:32:58,085 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 07:32:58,085 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:32:58,085 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:32:58,085 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30787.45 MB 2025-02-14 07:32:58,085 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39179.69 MB 2025-02-14 07:32:58,085 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8392.24 MB 2025-02-14 07:32:58,085 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44224.74 MB 2025-02-14 07:32:58,085 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48398.07 MB 2025-02-14 07:32:58,085 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4173.33 MB 2025-02-14 07:32:58,085 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39179.69 MB 2025-02-14 07:32:58,247 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7909] 2025-02-14 07:32:58,248 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:32:58,248 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 07:32:58,249 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:32:58,249 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 07:32:58,254 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 07:32:58,255 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:32:58,255 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 07:32:58,255 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 07:33:09,523 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:33:09,524 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 07:33:09,529 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 07:33:09,532 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:33:09,532 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 206, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 07:33:09,533 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:33:09,533 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 206, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 07:33:12,822 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 07:33:12,823 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 07:33:12,823 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.29 seconds 2025-02-14 07:33:12,823 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:33:12,823 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22740.97 MB 2025-02-14 07:33:12,823 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23469.99 MB 2025-02-14 07:33:12,823 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 729.02 MB 2025-02-14 07:33:12,823 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56740.54 MB 2025-02-14 07:33:12,823 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26275.22 MB 2025-02-14 07:33:12,823 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30465.33 MB 2025-02-14 07:33:12,823 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32439.64 MB 2025-02-14 07:33:12,838 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 07:33:12,838 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 07:33:12,838 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:33:12,838 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:33:12,838 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23469.99 MB 2025-02-14 07:33:12,838 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23823.92 MB 2025-02-14 07:33:12,838 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 353.93 MB 2025-02-14 07:33:12,838 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26275.22 MB 2025-02-14 07:33:12,838 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28443.67 MB 2025-02-14 07:33:12,838 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2168.46 MB 2025-02-14 07:33:12,838 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26414.59 MB 2025-02-14 07:33:13,854 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 07:33:13,854 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 07:33:13,854 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.01 seconds 2025-02-14 07:33:13,854 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:33:13,854 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23823.92 MB 2025-02-14 07:33:13,854 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24097.30 MB 2025-02-14 07:33:13,854 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 273.38 MB 2025-02-14 07:33:13,854 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28443.67 MB 2025-02-14 07:33:13,854 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25721.57 MB 2025-02-14 07:33:13,854 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2722.10 MB 2025-02-14 07:33:13,854 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28080.58 MB 2025-02-14 07:33:13,863 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 07:33:13,863 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 07:33:13,863 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:33:13,863 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:33:13,863 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24097.30 MB 2025-02-14 07:33:13,863 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25071.22 MB 2025-02-14 07:33:13,863 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 973.92 MB 2025-02-14 07:33:13,863 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25721.57 MB 2025-02-14 07:33:13,863 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27669.82 MB 2025-02-14 07:33:13,863 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1948.25 MB 2025-02-14 07:33:13,863 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25801.99 MB 2025-02-14 07:33:13,974 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 07:33:13,974 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 07:33:13,974 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 07:33:13,974 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:33:13,974 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25071.22 MB 2025-02-14 07:33:13,974 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26226.86 MB 2025-02-14 07:33:13,974 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1155.64 MB 2025-02-14 07:33:13,974 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27669.82 MB 2025-02-14 07:33:13,974 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30832.33 MB 2025-02-14 07:33:13,974 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3162.51 MB 2025-02-14 07:33:13,974 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29085.02 MB 2025-02-14 07:33:13,975 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 07:33:13,975 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 07:33:13,975 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 07:33:13,975 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:33:13,975 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24097.30 MB 2025-02-14 07:33:13,975 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26226.86 MB 2025-02-14 07:33:13,975 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2129.56 MB 2025-02-14 07:33:13,975 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25721.57 MB 2025-02-14 07:33:13,975 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30832.33 MB 2025-02-14 07:33:13,975 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5110.76 MB 2025-02-14 07:33:13,975 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29085.02 MB 2025-02-14 07:33:14,208 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 07:33:14,208 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 07:33:14,208 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 07:33:14,208 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:33:14,208 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27016.63 MB 2025-02-14 07:33:14,208 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19076.03 MB 2025-02-14 07:33:14,208 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7940.60 MB 2025-02-14 07:33:14,208 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30832.33 MB 2025-02-14 07:33:14,208 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30870.08 MB 2025-02-14 07:33:14,208 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 37.75 MB 2025-02-14 07:33:14,208 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27221.11 MB 2025-02-14 07:33:14,219 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 07:33:14,219 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 07:33:14,219 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:33:14,219 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:33:14,219 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19288.67 MB 2025-02-14 07:33:14,219 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19505.87 MB 2025-02-14 07:33:14,219 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 217.20 MB 2025-02-14 07:33:14,219 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30870.08 MB 2025-02-14 07:33:14,219 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30870.08 MB 2025-02-14 07:33:14,219 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:33:14,219 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19544.62 MB 2025-02-14 07:33:14,221 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 07:33:14,221 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 07:33:14,221 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.69 seconds 2025-02-14 07:33:14,221 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:33:14,221 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22023.25 MB 2025-02-14 07:33:14,221 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19706.95 MB 2025-02-14 07:33:14,221 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2316.30 MB 2025-02-14 07:33:14,221 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56740.54 MB 2025-02-14 07:33:14,221 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30870.08 MB 2025-02-14 07:33:14,221 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25870.47 MB 2025-02-14 07:33:14,221 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19706.95 MB 2025-02-14 07:33:14,489 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 07:33:14,489 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 07:33:14,489 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 07:33:14,489 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:33:14,489 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19706.95 MB 2025-02-14 07:33:14,489 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17776.48 MB 2025-02-14 07:33:14,489 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1930.46 MB 2025-02-14 07:33:14,489 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30870.08 MB 2025-02-14 07:33:14,489 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30870.08 MB 2025-02-14 07:33:14,489 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:33:14,489 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19706.95 MB 2025-02-14 07:33:14,507 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 07:33:14,508 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 07:33:14,514 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 07:33:14,514 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 07:33:14,514 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:33:14,514 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:33:14,514 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17776.48 MB 2025-02-14 07:33:14,514 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26215.50 MB 2025-02-14 07:33:14,514 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 07:33:14,514 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30870.08 MB 2025-02-14 07:33:14,514 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39260.78 MB 2025-02-14 07:33:14,514 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 07:33:14,514 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26215.50 MB 2025-02-14 07:33:14,678 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 07:33:14,679 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:33:14,679 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 07:33:14,680 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:33:14,680 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 07:33:14,685 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 07:33:14,686 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:33:14,686 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 07:33:14,686 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 07:34:56,919 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:34:56,920 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 07:34:56,925 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 07:34:56,929 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:34:56,929 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 204, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 07:34:56,930 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:34:56,930 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 204, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 07:35:00,172 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 07:35:00,173 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 07:35:00,173 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.24 seconds 2025-02-14 07:35:00,173 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:35:00,173 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14390.21 MB 2025-02-14 07:35:00,173 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15112.16 MB 2025-02-14 07:35:00,173 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 721.94 MB 2025-02-14 07:35:00,173 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51845.79 MB 2025-02-14 07:35:00,173 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17737.71 MB 2025-02-14 07:35:00,173 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34108.08 MB 2025-02-14 07:35:00,173 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24088.88 MB 2025-02-14 07:35:00,193 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 07:35:00,194 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 07:35:00,194 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:35:00,194 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:35:00,194 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15112.16 MB 2025-02-14 07:35:00,194 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15385.34 MB 2025-02-14 07:35:00,194 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 273.18 MB 2025-02-14 07:35:00,194 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17737.71 MB 2025-02-14 07:35:00,194 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19105.05 MB 2025-02-14 07:35:00,194 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1367.34 MB 2025-02-14 07:35:00,194 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17866.89 MB 2025-02-14 07:35:01,167 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 07:35:01,167 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 07:35:01,167 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.97 seconds 2025-02-14 07:35:01,167 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:35:01,167 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15385.34 MB 2025-02-14 07:35:01,168 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15641.40 MB 2025-02-14 07:35:01,168 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 256.07 MB 2025-02-14 07:35:01,168 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19105.05 MB 2025-02-14 07:35:01,168 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18421.38 MB 2025-02-14 07:35:01,168 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -683.67 MB 2025-02-14 07:35:01,168 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19641.68 MB 2025-02-14 07:35:01,179 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 07:35:01,179 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 07:35:01,179 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:35:01,179 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:35:01,179 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15641.40 MB 2025-02-14 07:35:01,179 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16552.88 MB 2025-02-14 07:35:01,179 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 911.48 MB 2025-02-14 07:35:01,179 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18421.38 MB 2025-02-14 07:35:01,179 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18878.56 MB 2025-02-14 07:35:01,179 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 457.18 MB 2025-02-14 07:35:01,179 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17236.80 MB 2025-02-14 07:35:01,320 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 07:35:01,321 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 07:35:01,321 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-14 07:35:01,321 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:35:01,321 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16552.88 MB 2025-02-14 07:35:01,321 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17635.10 MB 2025-02-14 07:35:01,321 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1082.22 MB 2025-02-14 07:35:01,321 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18878.56 MB 2025-02-14 07:35:01,321 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21621.64 MB 2025-02-14 07:35:01,321 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2743.07 MB 2025-02-14 07:35:01,321 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20310.19 MB 2025-02-14 07:35:01,322 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 07:35:01,322 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 07:35:01,322 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 07:35:01,322 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:35:01,322 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15641.40 MB 2025-02-14 07:35:01,322 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17635.10 MB 2025-02-14 07:35:01,322 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1993.70 MB 2025-02-14 07:35:01,322 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18421.38 MB 2025-02-14 07:35:01,322 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21621.64 MB 2025-02-14 07:35:01,322 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3200.25 MB 2025-02-14 07:35:01,322 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20310.19 MB 2025-02-14 07:35:01,465 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 07:35:01,465 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 07:35:01,465 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-14 07:35:01,465 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:35:01,465 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18375.04 MB 2025-02-14 07:35:01,465 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18745.12 MB 2025-02-14 07:35:01,465 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 370.08 MB 2025-02-14 07:35:01,465 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21621.64 MB 2025-02-14 07:35:01,465 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21816.67 MB 2025-02-14 07:35:01,465 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 195.04 MB 2025-02-14 07:35:01,465 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19090.80 MB 2025-02-14 07:35:01,483 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 07:35:01,483 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 07:35:01,483 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:35:01,483 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:35:01,483 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18944.34 MB 2025-02-14 07:35:01,483 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19171.17 MB 2025-02-14 07:35:01,483 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.83 MB 2025-02-14 07:35:01,483 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21816.67 MB 2025-02-14 07:35:01,483 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21818.77 MB 2025-02-14 07:35:01,483 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-14 07:35:01,483 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19208.54 MB 2025-02-14 07:35:01,485 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 07:35:01,486 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 07:35:01,486 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.55 seconds 2025-02-14 07:35:01,486 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:35:01,486 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13679.46 MB 2025-02-14 07:35:01,486 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19372.24 MB 2025-02-14 07:35:01,486 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5692.79 MB 2025-02-14 07:35:01,486 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51845.79 MB 2025-02-14 07:35:01,486 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21818.77 MB 2025-02-14 07:35:01,486 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30027.02 MB 2025-02-14 07:35:01,486 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19372.24 MB 2025-02-14 07:35:01,779 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 07:35:01,779 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 07:35:01,779 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-14 07:35:01,779 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:35:01,779 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19372.24 MB 2025-02-14 07:35:01,779 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17706.69 MB 2025-02-14 07:35:01,779 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1665.55 MB 2025-02-14 07:35:01,779 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21818.77 MB 2025-02-14 07:35:01,779 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21818.77 MB 2025-02-14 07:35:01,779 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:35:01,779 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19372.24 MB 2025-02-14 07:35:01,799 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 07:35:01,799 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 07:35:01,807 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 07:35:01,807 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 07:35:01,807 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:35:01,807 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:35:01,807 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17706.69 MB 2025-02-14 07:35:01,807 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26145.72 MB 2025-02-14 07:35:01,807 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 07:35:01,807 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21818.77 MB 2025-02-14 07:35:01,807 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32308.72 MB 2025-02-14 07:35:01,807 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 07:35:01,807 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26145.72 MB 2025-02-14 07:35:02,066 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 07:35:02,069 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:35:02,069 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 07:35:02,071 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:35:02,071 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 07:35:02,079 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 07:35:02,081 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:35:02,081 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 07:35:02,081 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 07:35:31,509 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:35:31,509 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 07:35:31,514 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 07:35:31,517 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:35:31,517 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2390, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 07:35:31,518 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:35:31,518 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2390, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 07:36:08,494 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 07:36:08,494 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 07:36:08,494 - resource_logging.py:150 - __exit__ - DEBUG - Time: 36.97 seconds 2025-02-14 07:36:08,494 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:36:08,494 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29624.44 MB 2025-02-14 07:36:08,494 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38082.52 MB 2025-02-14 07:36:08,494 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8458.08 MB 2025-02-14 07:36:08,494 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53353.64 MB 2025-02-14 07:36:08,494 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42215.67 MB 2025-02-14 07:36:08,494 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11137.97 MB 2025-02-14 07:36:08,494 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47023.86 MB 2025-02-14 07:36:08,652 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 07:36:08,653 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 07:36:08,653 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 07:36:08,653 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:36:08,653 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38082.52 MB 2025-02-14 07:36:08,653 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28203.63 MB 2025-02-14 07:36:08,653 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9878.89 MB 2025-02-14 07:36:08,653 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42215.67 MB 2025-02-14 07:36:08,653 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 72211.23 MB 2025-02-14 07:36:08,653 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 29995.57 MB 2025-02-14 07:36:08,653 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 61314.31 MB 2025-02-14 07:36:10,638 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 07:36:10,638 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 07:36:10,638 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.98 seconds 2025-02-14 07:36:10,638 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:36:10,638 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28203.63 MB 2025-02-14 07:36:10,638 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28734.47 MB 2025-02-14 07:36:10,638 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 07:36:10,638 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 72211.23 MB 2025-02-14 07:36:10,638 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30941.38 MB 2025-02-14 07:36:10,638 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -41269.85 MB 2025-02-14 07:36:10,638 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32714.85 MB 2025-02-14 07:36:10,652 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 07:36:10,652 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 07:36:10,652 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:36:10,652 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:36:10,652 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28734.47 MB 2025-02-14 07:36:10,652 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30624.01 MB 2025-02-14 07:36:10,652 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 07:36:10,652 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30941.38 MB 2025-02-14 07:36:10,652 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34244.40 MB 2025-02-14 07:36:10,652 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 07:36:10,652 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32041.44 MB 2025-02-14 07:36:10,864 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 07:36:10,864 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 07:36:10,864 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 07:36:10,864 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:36:10,864 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30624.01 MB 2025-02-14 07:36:10,864 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32865.86 MB 2025-02-14 07:36:10,864 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 07:36:10,864 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34244.40 MB 2025-02-14 07:36:10,864 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40850.42 MB 2025-02-14 07:36:10,864 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 07:36:10,864 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38410.15 MB 2025-02-14 07:36:10,865 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 07:36:10,865 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 07:36:10,865 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 07:36:10,865 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:36:10,865 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28734.47 MB 2025-02-14 07:36:10,865 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32865.86 MB 2025-02-14 07:36:10,865 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 07:36:10,865 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30941.38 MB 2025-02-14 07:36:10,865 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40850.42 MB 2025-02-14 07:36:10,865 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9909.04 MB 2025-02-14 07:36:10,865 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38410.15 MB 2025-02-14 07:36:11,032 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 07:36:11,032 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 07:36:11,032 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 07:36:11,032 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:36:11,032 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34399.41 MB 2025-02-14 07:36:11,032 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35166.41 MB 2025-02-14 07:36:11,032 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 07:36:11,032 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40850.42 MB 2025-02-14 07:36:11,032 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41267.76 MB 2025-02-14 07:36:11,032 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 07:36:11,032 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35874.20 MB 2025-02-14 07:36:11,051 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 07:36:11,051 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 07:36:11,051 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:36:11,051 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:36:11,051 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35579.30 MB 2025-02-14 07:36:11,051 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35808.14 MB 2025-02-14 07:36:11,051 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.84 MB 2025-02-14 07:36:11,051 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41267.76 MB 2025-02-14 07:36:11,051 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41267.76 MB 2025-02-14 07:36:11,051 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:36:11,051 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36012.80 MB 2025-02-14 07:36:11,052 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 07:36:11,052 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 07:36:11,052 - resource_logging.py:150 - __exit__ - DEBUG - Time: 39.53 seconds 2025-02-14 07:36:11,052 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:36:11,052 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21296.57 MB 2025-02-14 07:36:11,052 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36008.89 MB 2025-02-14 07:36:11,052 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14712.31 MB 2025-02-14 07:36:11,052 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49123.69 MB 2025-02-14 07:36:11,052 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41267.76 MB 2025-02-14 07:36:11,052 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7855.93 MB 2025-02-14 07:36:11,052 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36012.80 MB 2025-02-14 07:36:11,323 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 07:36:11,323 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 07:36:11,323 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 07:36:11,323 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:36:11,323 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36008.89 MB 2025-02-14 07:36:11,323 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26296.01 MB 2025-02-14 07:36:11,323 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9712.88 MB 2025-02-14 07:36:11,323 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41267.76 MB 2025-02-14 07:36:11,323 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41267.76 MB 2025-02-14 07:36:11,323 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:36:11,323 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37914.72 MB 2025-02-14 07:36:11,342 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8149, cut from 8151 2025-02-14 07:36:11,342 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 07:36:11,348 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 07:36:11,348 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 07:36:11,348 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:36:11,348 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:36:11,348 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26296.01 MB 2025-02-14 07:36:11,349 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34721.48 MB 2025-02-14 07:36:11,349 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8425.47 MB 2025-02-14 07:36:11,349 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41267.76 MB 2025-02-14 07:36:11,349 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45455.77 MB 2025-02-14 07:36:11,349 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4188.01 MB 2025-02-14 07:36:11,349 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34721.48 MB 2025-02-14 07:36:11,511 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7941] 2025-02-14 07:36:11,512 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:36:11,512 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 07:36:11,513 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:36:11,513 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 07:36:11,518 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 07:36:11,519 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:36:11,519 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 07:36:11,519 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 07:37:24,231 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:37:24,231 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 07:37:24,236 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 07:37:24,240 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:37:24,240 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 712, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 07:37:24,241 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:37:24,241 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 712, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 07:37:35,229 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 07:37:35,229 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 07:37:35,229 - resource_logging.py:150 - __exit__ - DEBUG - Time: 10.98 seconds 2025-02-14 07:37:35,229 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:37:35,229 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17930.04 MB 2025-02-14 07:37:35,229 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20450.81 MB 2025-02-14 07:37:35,229 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2520.78 MB 2025-02-14 07:37:35,229 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53831.79 MB 2025-02-14 07:37:35,229 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25033.70 MB 2025-02-14 07:37:35,229 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28798.09 MB 2025-02-14 07:37:35,229 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29439.84 MB 2025-02-14 07:37:35,275 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 07:37:35,275 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 07:37:35,275 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-14 07:37:35,275 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:37:35,275 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20450.81 MB 2025-02-14 07:37:35,276 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19480.37 MB 2025-02-14 07:37:35,276 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -970.45 MB 2025-02-14 07:37:35,276 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25033.70 MB 2025-02-14 07:37:35,276 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32642.17 MB 2025-02-14 07:37:35,276 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7608.47 MB 2025-02-14 07:37:35,276 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29580.90 MB 2025-02-14 07:37:37,185 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 07:37:37,185 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 07:37:37,185 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 07:37:37,185 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:37:37,185 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19480.37 MB 2025-02-14 07:37:37,185 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20011.21 MB 2025-02-14 07:37:37,185 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 07:37:37,185 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32642.17 MB 2025-02-14 07:37:37,185 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24637.34 MB 2025-02-14 07:37:37,185 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8004.83 MB 2025-02-14 07:37:37,185 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23990.54 MB 2025-02-14 07:37:37,199 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 07:37:37,199 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 07:37:37,199 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:37:37,199 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:37:37,199 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20011.21 MB 2025-02-14 07:37:37,199 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21900.74 MB 2025-02-14 07:37:37,199 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 07:37:37,199 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24637.34 MB 2025-02-14 07:37:37,199 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26524.78 MB 2025-02-14 07:37:37,199 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 07:37:37,199 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23318.17 MB 2025-02-14 07:37:37,410 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 07:37:37,410 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 07:37:37,410 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 07:37:37,410 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:37:37,410 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21900.74 MB 2025-02-14 07:37:37,410 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24142.60 MB 2025-02-14 07:37:37,410 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 07:37:37,410 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26524.78 MB 2025-02-14 07:37:37,410 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32187.09 MB 2025-02-14 07:37:37,410 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 07:37:37,410 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29686.88 MB 2025-02-14 07:37:37,411 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 07:37:37,411 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 07:37:37,411 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 07:37:37,411 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:37:37,411 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20011.21 MB 2025-02-14 07:37:37,411 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24142.60 MB 2025-02-14 07:37:37,411 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 07:37:37,411 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24637.34 MB 2025-02-14 07:37:37,411 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32187.09 MB 2025-02-14 07:37:37,411 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-14 07:37:37,411 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29686.88 MB 2025-02-14 07:37:37,583 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 07:37:37,583 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 07:37:37,583 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 07:37:37,584 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:37:37,584 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25676.14 MB 2025-02-14 07:37:37,584 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26443.14 MB 2025-02-14 07:37:37,584 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 07:37:37,584 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32187.09 MB 2025-02-14 07:37:37,584 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32602.32 MB 2025-02-14 07:37:37,584 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 07:37:37,584 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27150.93 MB 2025-02-14 07:37:37,605 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 07:37:37,605 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 07:37:37,605 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:37:37,605 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:37:37,605 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26856.03 MB 2025-02-14 07:37:37,605 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27085.65 MB 2025-02-14 07:37:37,605 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.62 MB 2025-02-14 07:37:37,605 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32602.32 MB 2025-02-14 07:37:37,605 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32602.32 MB 2025-02-14 07:37:37,605 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:37:37,605 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27319.66 MB 2025-02-14 07:37:37,606 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 07:37:37,606 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 07:37:37,606 - resource_logging.py:150 - __exit__ - DEBUG - Time: 13.36 seconds 2025-02-14 07:37:37,606 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:37:37,606 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15449.37 MB 2025-02-14 07:37:37,606 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27286.73 MB 2025-02-14 07:37:37,606 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11837.36 MB 2025-02-14 07:37:37,606 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53831.79 MB 2025-02-14 07:37:37,606 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32602.32 MB 2025-02-14 07:37:37,606 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21229.47 MB 2025-02-14 07:37:37,606 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27319.66 MB 2025-02-14 07:37:37,873 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 07:37:37,873 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 07:37:37,873 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 07:37:37,873 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:37:37,873 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27286.73 MB 2025-02-14 07:37:37,873 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20453.76 MB 2025-02-14 07:37:37,873 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6832.97 MB 2025-02-14 07:37:37,873 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32602.32 MB 2025-02-14 07:37:37,873 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32602.32 MB 2025-02-14 07:37:37,873 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:37:37,873 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29798.39 MB 2025-02-14 07:37:37,891 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 07:37:37,892 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 07:37:37,898 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 07:37:37,898 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 07:37:37,898 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:37:37,898 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:37:37,898 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20453.76 MB 2025-02-14 07:37:37,898 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28892.78 MB 2025-02-14 07:37:37,898 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 07:37:37,898 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32602.32 MB 2025-02-14 07:37:37,898 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40993.03 MB 2025-02-14 07:37:37,898 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 07:37:37,898 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28892.78 MB 2025-02-14 07:37:38,061 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 07:37:38,063 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:37:38,063 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 07:37:38,064 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:37:38,064 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 07:37:38,069 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 07:37:38,070 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:37:38,070 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 07:37:38,070 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 07:37:53,478 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:37:53,478 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 07:37:53,483 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 07:37:53,487 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:37:53,487 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2004, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 07:37:53,488 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:37:53,489 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2004, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 07:38:24,736 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 07:38:24,736 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 07:38:24,736 - resource_logging.py:150 - __exit__ - DEBUG - Time: 31.24 seconds 2025-02-14 07:38:24,736 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:38:24,736 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26932.90 MB 2025-02-14 07:38:24,736 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34025.47 MB 2025-02-14 07:38:24,736 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7092.57 MB 2025-02-14 07:38:24,736 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53578.04 MB 2025-02-14 07:38:24,737 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40772.83 MB 2025-02-14 07:38:24,737 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12805.21 MB 2025-02-14 07:38:24,737 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42973.36 MB 2025-02-14 07:38:24,802 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 07:38:24,802 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 07:38:24,802 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 07:38:24,802 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:38:24,802 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34025.47 MB 2025-02-14 07:38:24,803 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24426.22 MB 2025-02-14 07:38:24,803 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9599.25 MB 2025-02-14 07:38:24,803 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40772.83 MB 2025-02-14 07:38:24,803 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40772.83 MB 2025-02-14 07:38:24,803 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:38:24,803 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36118.44 MB 2025-02-14 07:38:25,526 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 07:38:25,526 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 07:38:25,526 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.72 seconds 2025-02-14 07:38:25,526 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:38:25,526 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24426.22 MB 2025-02-14 07:38:25,526 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24622.63 MB 2025-02-14 07:38:25,526 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 196.41 MB 2025-02-14 07:38:25,526 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40772.83 MB 2025-02-14 07:38:25,526 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29492.25 MB 2025-02-14 07:38:25,526 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11280.58 MB 2025-02-14 07:38:25,526 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28596.66 MB 2025-02-14 07:38:25,533 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 07:38:25,533 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 07:38:25,533 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 07:38:25,533 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:38:25,533 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24622.63 MB 2025-02-14 07:38:25,533 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25321.59 MB 2025-02-14 07:38:25,533 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 698.96 MB 2025-02-14 07:38:25,533 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29492.25 MB 2025-02-14 07:38:25,533 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29492.25 MB 2025-02-14 07:38:25,533 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:38:25,533 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25846.04 MB 2025-02-14 07:38:25,614 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 07:38:25,614 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 07:38:25,614 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 07:38:25,614 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:38:25,614 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25321.59 MB 2025-02-14 07:38:25,614 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26151.12 MB 2025-02-14 07:38:25,614 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 829.53 MB 2025-02-14 07:38:25,614 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29492.25 MB 2025-02-14 07:38:25,614 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29492.25 MB 2025-02-14 07:38:25,614 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:38:25,614 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28202.46 MB 2025-02-14 07:38:25,614 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 07:38:25,614 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 07:38:25,614 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 07:38:25,615 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:38:25,615 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24622.63 MB 2025-02-14 07:38:25,615 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26151.12 MB 2025-02-14 07:38:25,615 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1528.49 MB 2025-02-14 07:38:25,615 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29492.25 MB 2025-02-14 07:38:25,615 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29492.25 MB 2025-02-14 07:38:25,615 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:38:25,615 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28202.46 MB 2025-02-14 07:38:25,679 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 07:38:25,679 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 07:38:25,679 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 07:38:25,679 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:38:25,679 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26718.53 MB 2025-02-14 07:38:25,679 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27002.32 MB 2025-02-14 07:38:25,679 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 283.79 MB 2025-02-14 07:38:25,679 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29492.25 MB 2025-02-14 07:38:25,679 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29643.24 MB 2025-02-14 07:38:25,679 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 150.99 MB 2025-02-14 07:38:25,679 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27274.42 MB 2025-02-14 07:38:25,688 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 07:38:25,689 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 07:38:25,689 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:38:25,689 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:38:25,689 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27155.09 MB 2025-02-14 07:38:25,689 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27249.07 MB 2025-02-14 07:38:25,689 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 93.98 MB 2025-02-14 07:38:25,689 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29643.24 MB 2025-02-14 07:38:25,689 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29645.34 MB 2025-02-14 07:38:25,689 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-14 07:38:25,689 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27249.07 MB 2025-02-14 07:38:25,690 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 07:38:25,690 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 07:38:25,690 - resource_logging.py:150 - __exit__ - DEBUG - Time: 32.20 seconds 2025-02-14 07:38:25,690 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:38:25,690 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19950.80 MB 2025-02-14 07:38:25,690 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27332.41 MB 2025-02-14 07:38:25,690 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7381.61 MB 2025-02-14 07:38:25,690 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53578.04 MB 2025-02-14 07:38:25,690 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29645.34 MB 2025-02-14 07:38:25,690 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23932.70 MB 2025-02-14 07:38:25,690 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27332.41 MB 2025-02-14 07:38:25,798 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 07:38:25,798 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 07:38:25,798 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 07:38:25,798 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:38:25,798 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27332.41 MB 2025-02-14 07:38:25,798 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21942.32 MB 2025-02-14 07:38:25,798 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5390.09 MB 2025-02-14 07:38:25,798 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29645.34 MB 2025-02-14 07:38:25,798 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29645.34 MB 2025-02-14 07:38:25,798 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:38:25,798 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27790.50 MB 2025-02-14 07:38:25,806 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 3375, cut from 3377 2025-02-14 07:38:25,807 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 1 ('] 2025-02-14 07:38:25,810 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 07:38:25,810 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 07:38:25,810 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:38:25,810 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:38:25,810 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21942.32 MB 2025-02-14 07:38:25,810 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25439.91 MB 2025-02-14 07:38:25,810 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3497.59 MB 2025-02-14 07:38:25,810 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29645.34 MB 2025-02-14 07:38:25,810 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29645.34 MB 2025-02-14 07:38:25,810 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:38:25,810 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25439.91 MB 2025-02-14 07:38:25,879 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 3167] 2025-02-14 07:38:25,880 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:38:25,880 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 07:38:25,881 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:38:25,881 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 07:38:25,886 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 07:38:25,887 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:38:25,887 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 07:38:25,887 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 1 ('] 2025-02-14 07:39:28,946 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:39:28,946 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 07:39:28,953 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 07:39:28,959 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:39:28,959 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 275, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 07:39:28,961 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:39:28,961 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 275, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 07:39:33,281 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 07:39:33,281 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 07:39:33,281 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.31 seconds 2025-02-14 07:39:33,281 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:39:33,281 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14884.95 MB 2025-02-14 07:39:33,281 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15858.16 MB 2025-02-14 07:39:33,281 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 973.21 MB 2025-02-14 07:39:33,281 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33122.42 MB 2025-02-14 07:39:33,281 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25297.94 MB 2025-02-14 07:39:33,281 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7824.47 MB 2025-02-14 07:39:33,281 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24809.31 MB 2025-02-14 07:39:33,307 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 07:39:33,307 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 07:39:33,307 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:39:33,307 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:39:33,307 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15858.16 MB 2025-02-14 07:39:33,307 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16315.57 MB 2025-02-14 07:39:33,307 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 457.41 MB 2025-02-14 07:39:33,307 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25297.94 MB 2025-02-14 07:39:33,307 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25297.94 MB 2025-02-14 07:39:33,307 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:39:33,307 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19703.35 MB 2025-02-14 07:39:34,653 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 07:39:34,654 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 07:39:34,654 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.34 seconds 2025-02-14 07:39:34,654 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:39:34,654 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16315.57 MB 2025-02-14 07:39:34,654 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16677.87 MB 2025-02-14 07:39:34,654 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 362.30 MB 2025-02-14 07:39:34,654 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25297.94 MB 2025-02-14 07:39:34,654 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25297.94 MB 2025-02-14 07:39:34,654 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:39:34,654 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20655.87 MB 2025-02-14 07:39:34,667 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 07:39:34,667 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 07:39:34,667 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:39:34,667 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:39:34,667 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16677.87 MB 2025-02-14 07:39:34,667 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17967.18 MB 2025-02-14 07:39:34,667 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1289.31 MB 2025-02-14 07:39:34,667 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25297.94 MB 2025-02-14 07:39:34,667 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25297.94 MB 2025-02-14 07:39:34,667 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:39:34,668 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18934.58 MB 2025-02-14 07:39:34,857 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 07:39:34,857 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 07:39:34,857 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.19 seconds 2025-02-14 07:39:34,857 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:39:34,857 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17967.18 MB 2025-02-14 07:39:34,858 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19497.92 MB 2025-02-14 07:39:34,858 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1530.74 MB 2025-02-14 07:39:34,858 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25297.94 MB 2025-02-14 07:39:34,858 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25297.94 MB 2025-02-14 07:39:34,858 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:39:34,858 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23281.88 MB 2025-02-14 07:39:34,859 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 07:39:34,859 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 07:39:34,859 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 07:39:34,859 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:39:34,859 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16677.87 MB 2025-02-14 07:39:34,859 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19497.92 MB 2025-02-14 07:39:34,859 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2820.06 MB 2025-02-14 07:39:34,859 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25297.94 MB 2025-02-14 07:39:34,859 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25297.94 MB 2025-02-14 07:39:34,859 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:39:34,859 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23281.88 MB 2025-02-14 07:39:34,989 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 07:39:34,989 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 07:39:34,989 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 07:39:34,989 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:39:34,989 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20544.57 MB 2025-02-14 07:39:34,989 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21068.05 MB 2025-02-14 07:39:34,989 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 523.48 MB 2025-02-14 07:39:34,989 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25297.94 MB 2025-02-14 07:39:34,989 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25578.96 MB 2025-02-14 07:39:34,989 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 281.02 MB 2025-02-14 07:39:34,989 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21551.11 MB 2025-02-14 07:39:35,003 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 07:39:35,003 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 07:39:35,003 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:39:35,003 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:39:35,003 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21349.85 MB 2025-02-14 07:39:35,003 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21578.74 MB 2025-02-14 07:39:35,003 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.89 MB 2025-02-14 07:39:35,003 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25578.96 MB 2025-02-14 07:39:35,003 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25578.96 MB 2025-02-14 07:39:35,003 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:39:35,003 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21701.61 MB 2025-02-14 07:39:35,004 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 07:39:35,004 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 07:39:35,004 - resource_logging.py:150 - __exit__ - DEBUG - Time: 6.04 seconds 2025-02-14 07:39:35,004 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:39:35,004 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13926.83 MB 2025-02-14 07:39:35,004 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21779.81 MB 2025-02-14 07:39:35,004 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7852.98 MB 2025-02-14 07:39:35,004 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33122.42 MB 2025-02-14 07:39:35,004 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25578.96 MB 2025-02-14 07:39:35,004 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7543.46 MB 2025-02-14 07:39:35,004 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21779.81 MB 2025-02-14 07:39:35,272 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 07:39:35,272 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 07:39:35,272 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 07:39:35,272 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:39:35,272 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21779.81 MB 2025-02-14 07:39:35,272 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24793.84 MB 2025-02-14 07:39:35,272 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-14 07:39:35,272 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25578.96 MB 2025-02-14 07:39:35,272 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25981.62 MB 2025-02-14 07:39:35,272 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 402.65 MB 2025-02-14 07:39:35,272 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25095.47 MB 2025-02-14 07:39:35,290 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 07:39:35,290 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 07:39:35,296 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 07:39:35,296 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 07:39:35,296 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:39:35,296 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:39:35,296 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18331.87 MB 2025-02-14 07:39:35,296 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26770.89 MB 2025-02-14 07:39:35,296 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 07:39:35,296 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25981.62 MB 2025-02-14 07:39:35,296 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36471.57 MB 2025-02-14 07:39:35,296 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 07:39:35,296 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26770.89 MB 2025-02-14 07:39:35,458 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 07:39:35,460 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:39:35,460 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 07:39:35,461 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:39:35,461 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 07:39:35,465 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 07:39:35,466 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:39:35,466 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 07:39:35,466 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 07:39:45,740 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:39:45,740 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 07:39:45,745 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 07:39:45,748 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:39:45,749 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1345, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 07:39:45,749 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:39:45,749 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1345, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 07:40:06,600 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 07:40:06,600 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 07:40:06,600 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.84 seconds 2025-02-14 07:40:06,600 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:40:06,600 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22340.88 MB 2025-02-14 07:40:06,600 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27101.42 MB 2025-02-14 07:40:06,600 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4760.54 MB 2025-02-14 07:40:06,600 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49056.58 MB 2025-02-14 07:40:06,601 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38447.09 MB 2025-02-14 07:40:06,601 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10609.49 MB 2025-02-14 07:40:06,601 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36115.61 MB 2025-02-14 07:40:06,675 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 07:40:06,675 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 07:40:06,675 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 07:40:06,675 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:40:06,675 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27101.42 MB 2025-02-14 07:40:06,675 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22770.09 MB 2025-02-14 07:40:06,675 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4331.33 MB 2025-02-14 07:40:06,675 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38447.09 MB 2025-02-14 07:40:06,675 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47559.21 MB 2025-02-14 07:40:06,675 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9112.13 MB 2025-02-14 07:40:06,675 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40805.29 MB 2025-02-14 07:40:08,626 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 07:40:08,627 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 07:40:08,627 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.95 seconds 2025-02-14 07:40:08,627 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:40:08,627 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22770.09 MB 2025-02-14 07:40:08,627 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23300.93 MB 2025-02-14 07:40:08,627 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 07:40:08,627 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47559.21 MB 2025-02-14 07:40:08,627 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33686.55 MB 2025-02-14 07:40:08,627 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13872.66 MB 2025-02-14 07:40:08,627 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27280.26 MB 2025-02-14 07:40:08,640 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 07:40:08,640 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 07:40:08,640 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:40:08,640 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:40:08,640 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23300.93 MB 2025-02-14 07:40:08,640 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25190.46 MB 2025-02-14 07:40:08,640 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 07:40:08,640 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33686.55 MB 2025-02-14 07:40:08,640 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33686.55 MB 2025-02-14 07:40:08,640 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:40:08,640 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26607.89 MB 2025-02-14 07:40:08,844 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 07:40:08,844 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 07:40:08,844 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 07:40:08,844 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:40:08,844 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25190.46 MB 2025-02-14 07:40:08,844 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27433.76 MB 2025-02-14 07:40:08,844 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2243.29 MB 2025-02-14 07:40:08,844 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33686.55 MB 2025-02-14 07:40:08,844 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35573.99 MB 2025-02-14 07:40:08,844 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 07:40:08,844 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32978.04 MB 2025-02-14 07:40:08,845 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 07:40:08,845 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 07:40:08,845 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 07:40:08,845 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:40:08,845 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23300.93 MB 2025-02-14 07:40:08,845 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27433.76 MB 2025-02-14 07:40:08,845 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4132.83 MB 2025-02-14 07:40:08,845 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33686.55 MB 2025-02-14 07:40:08,845 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35573.99 MB 2025-02-14 07:40:08,845 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 07:40:08,845 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32978.04 MB 2025-02-14 07:40:09,015 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 07:40:09,016 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 07:40:09,016 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 07:40:09,016 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:40:09,016 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28967.30 MB 2025-02-14 07:40:09,016 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29734.30 MB 2025-02-14 07:40:09,016 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 07:40:09,016 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35573.99 MB 2025-02-14 07:40:09,016 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35989.23 MB 2025-02-14 07:40:09,016 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 07:40:09,016 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30442.09 MB 2025-02-14 07:40:09,034 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 07:40:09,034 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 07:40:09,034 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:40:09,034 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:40:09,034 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30147.19 MB 2025-02-14 07:40:09,034 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30375.90 MB 2025-02-14 07:40:09,034 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.72 MB 2025-02-14 07:40:09,034 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35989.23 MB 2025-02-14 07:40:09,034 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35989.23 MB 2025-02-14 07:40:09,034 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:40:09,034 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30598.97 MB 2025-02-14 07:40:09,036 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 07:40:09,036 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 07:40:09,036 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.28 seconds 2025-02-14 07:40:09,036 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:40:09,036 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17654.79 MB 2025-02-14 07:40:09,036 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30576.19 MB 2025-02-14 07:40:09,036 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12921.40 MB 2025-02-14 07:40:09,036 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49056.58 MB 2025-02-14 07:40:09,036 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35989.23 MB 2025-02-14 07:40:09,036 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13067.35 MB 2025-02-14 07:40:09,036 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30598.97 MB 2025-02-14 07:40:09,305 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 07:40:09,305 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 07:40:09,305 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 07:40:09,305 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:40:09,305 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30576.19 MB 2025-02-14 07:40:09,305 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22646.99 MB 2025-02-14 07:40:09,305 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7929.20 MB 2025-02-14 07:40:09,305 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35989.23 MB 2025-02-14 07:40:09,305 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35989.23 MB 2025-02-14 07:40:09,305 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:40:09,305 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33078.03 MB 2025-02-14 07:40:09,323 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8130, cut from 8132 2025-02-14 07:40:09,323 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-14 07:40:09,329 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 07:40:09,329 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 07:40:09,329 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:40:09,329 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:40:09,329 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22646.99 MB 2025-02-14 07:40:09,329 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31052.65 MB 2025-02-14 07:40:09,329 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8405.66 MB 2025-02-14 07:40:09,329 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35989.23 MB 2025-02-14 07:40:09,329 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40168.85 MB 2025-02-14 07:40:09,329 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4179.62 MB 2025-02-14 07:40:09,329 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31052.65 MB 2025-02-14 07:40:09,480 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7922] 2025-02-14 07:40:09,482 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:40:09,482 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 07:40:09,483 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:40:09,483 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 07:40:09,487 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 07:40:09,488 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:40:09,488 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 07:40:09,488 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-14 07:41:03,567 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:41:03,567 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 07:41:03,574 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 07:41:03,580 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:41:03,580 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 178, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 07:41:03,582 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:41:03,582 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 178, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 07:41:06,408 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 07:41:06,409 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 07:41:06,409 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.82 seconds 2025-02-14 07:41:06,409 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:41:06,409 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14209.04 MB 2025-02-14 07:41:06,409 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14838.97 MB 2025-02-14 07:41:06,409 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 629.93 MB 2025-02-14 07:41:06,409 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48528.10 MB 2025-02-14 07:41:06,409 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17850.96 MB 2025-02-14 07:41:06,409 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30677.14 MB 2025-02-14 07:41:06,409 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23681.22 MB 2025-02-14 07:41:06,428 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 07:41:06,428 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 07:41:06,428 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:41:06,428 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:41:06,428 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14838.97 MB 2025-02-14 07:41:06,428 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15137.80 MB 2025-02-14 07:41:06,428 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 298.83 MB 2025-02-14 07:41:06,428 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17850.96 MB 2025-02-14 07:41:06,428 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18478.01 MB 2025-02-14 07:41:06,428 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 627.05 MB 2025-02-14 07:41:06,428 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17332.92 MB 2025-02-14 07:41:07,311 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 07:41:07,312 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 07:41:07,312 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.88 seconds 2025-02-14 07:41:07,312 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:41:07,312 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15137.80 MB 2025-02-14 07:41:07,312 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15372.70 MB 2025-02-14 07:41:07,312 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 234.90 MB 2025-02-14 07:41:07,312 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18478.01 MB 2025-02-14 07:41:07,312 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17842.57 MB 2025-02-14 07:41:07,312 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -635.44 MB 2025-02-14 07:41:07,312 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19309.28 MB 2025-02-14 07:41:07,324 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 07:41:07,324 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 07:41:07,324 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:41:07,324 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:41:07,324 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15372.64 MB 2025-02-14 07:41:07,324 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16208.55 MB 2025-02-14 07:41:07,324 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 835.92 MB 2025-02-14 07:41:07,324 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17842.57 MB 2025-02-14 07:41:07,324 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18262.00 MB 2025-02-14 07:41:07,324 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 419.43 MB 2025-02-14 07:41:07,324 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16835.77 MB 2025-02-14 07:41:07,451 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 07:41:07,452 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 07:41:07,452 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 07:41:07,452 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:41:07,452 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16208.55 MB 2025-02-14 07:41:07,452 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17200.61 MB 2025-02-14 07:41:07,452 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 992.06 MB 2025-02-14 07:41:07,452 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18262.00 MB 2025-02-14 07:41:07,452 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20778.58 MB 2025-02-14 07:41:07,452 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2516.58 MB 2025-02-14 07:41:07,452 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19655.75 MB 2025-02-14 07:41:07,453 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 07:41:07,453 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 07:41:07,453 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-14 07:41:07,453 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:41:07,453 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15372.64 MB 2025-02-14 07:41:07,453 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17200.61 MB 2025-02-14 07:41:07,453 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1827.97 MB 2025-02-14 07:41:07,453 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17842.57 MB 2025-02-14 07:41:07,453 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20778.58 MB 2025-02-14 07:41:07,453 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2936.01 MB 2025-02-14 07:41:07,453 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19655.75 MB 2025-02-14 07:41:07,579 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 07:41:07,580 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 07:41:07,580 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 07:41:07,580 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:41:07,580 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17879.20 MB 2025-02-14 07:41:07,580 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18220.44 MB 2025-02-14 07:41:07,580 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 341.23 MB 2025-02-14 07:41:07,580 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20778.58 MB 2025-02-14 07:41:07,580 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20963.13 MB 2025-02-14 07:41:07,580 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 184.55 MB 2025-02-14 07:41:07,580 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18540.48 MB 2025-02-14 07:41:07,596 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 07:41:07,596 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 07:41:07,596 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:41:07,596 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:41:07,596 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18403.15 MB 2025-02-14 07:41:07,596 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18629.80 MB 2025-02-14 07:41:07,596 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.65 MB 2025-02-14 07:41:07,596 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20963.13 MB 2025-02-14 07:41:07,596 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20963.13 MB 2025-02-14 07:41:07,596 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:41:07,596 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18643.31 MB 2025-02-14 07:41:07,598 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 07:41:07,598 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 07:41:07,598 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.01 seconds 2025-02-14 07:41:07,598 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:41:07,599 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13588.87 MB 2025-02-14 07:41:07,599 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18830.43 MB 2025-02-14 07:41:07,599 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5241.56 MB 2025-02-14 07:41:07,599 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48528.10 MB 2025-02-14 07:41:07,599 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20963.13 MB 2025-02-14 07:41:07,599 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27564.97 MB 2025-02-14 07:41:07,599 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18830.43 MB 2025-02-14 07:41:07,886 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 07:41:07,886 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 07:41:07,886 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.28 seconds 2025-02-14 07:41:07,886 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:41:07,886 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18830.43 MB 2025-02-14 07:41:07,887 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17535.58 MB 2025-02-14 07:41:07,887 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1294.86 MB 2025-02-14 07:41:07,887 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20963.13 MB 2025-02-14 07:41:07,887 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20963.13 MB 2025-02-14 07:41:07,887 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:41:07,887 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19065.17 MB 2025-02-14 07:41:07,906 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8144, cut from 8146 2025-02-14 07:41:07,906 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 video rate for this video is 2,'] 2025-02-14 07:41:07,914 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 07:41:07,914 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 07:41:07,914 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:41:07,914 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:41:07,914 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17535.58 MB 2025-02-14 07:41:07,914 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25956.35 MB 2025-02-14 07:41:07,914 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8420.78 MB 2025-02-14 07:41:07,914 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20963.13 MB 2025-02-14 07:41:07,914 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31427.92 MB 2025-02-14 07:41:07,914 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10464.79 MB 2025-02-14 07:41:07,914 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25956.35 MB 2025-02-14 07:41:08,170 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7936] 2025-02-14 07:41:08,173 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:41:08,173 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 07:41:08,175 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:41:08,175 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 07:41:08,183 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 07:41:08,185 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:41:08,185 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 07:41:08,185 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 video rate for this video is 2,'] 2025-02-14 07:41:16,810 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:41:16,810 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 07:41:16,815 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 07:41:16,819 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:41:16,819 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1231, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 07:41:16,820 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:41:16,820 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1231, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 07:41:35,861 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 07:41:35,861 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 07:41:35,861 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.03 seconds 2025-02-14 07:41:35,861 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:41:35,861 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21546.51 MB 2025-02-14 07:41:35,861 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25902.95 MB 2025-02-14 07:41:35,861 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4356.44 MB 2025-02-14 07:41:35,861 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39799.75 MB 2025-02-14 07:41:35,861 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38010.88 MB 2025-02-14 07:41:35,861 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1788.87 MB 2025-02-14 07:41:35,861 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34868.25 MB 2025-02-14 07:41:35,932 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 07:41:35,932 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 07:41:35,932 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 07:41:35,932 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:41:35,932 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25902.95 MB 2025-02-14 07:41:35,932 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22177.44 MB 2025-02-14 07:41:35,932 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3725.52 MB 2025-02-14 07:41:35,932 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38010.88 MB 2025-02-14 07:41:35,932 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46540.00 MB 2025-02-14 07:41:35,932 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8529.12 MB 2025-02-14 07:41:35,932 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38780.63 MB 2025-02-14 07:41:37,859 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 07:41:37,859 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 07:41:37,859 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 07:41:37,859 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:41:37,859 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22177.44 MB 2025-02-14 07:41:37,859 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22708.28 MB 2025-02-14 07:41:37,859 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 07:41:37,859 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46540.00 MB 2025-02-14 07:41:37,859 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29467.08 MB 2025-02-14 07:41:37,859 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17072.91 MB 2025-02-14 07:41:37,859 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26687.61 MB 2025-02-14 07:41:37,872 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 07:41:37,872 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 07:41:37,872 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:41:37,872 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:41:37,872 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22708.28 MB 2025-02-14 07:41:37,872 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24597.81 MB 2025-02-14 07:41:37,872 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 07:41:37,872 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29467.08 MB 2025-02-14 07:41:37,872 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29467.08 MB 2025-02-14 07:41:37,872 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:41:37,872 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26015.24 MB 2025-02-14 07:41:38,085 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 07:41:38,085 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 07:41:38,085 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 07:41:38,085 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:41:38,085 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24597.81 MB 2025-02-14 07:41:38,085 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26839.67 MB 2025-02-14 07:41:38,085 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 07:41:38,085 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29467.08 MB 2025-02-14 07:41:38,085 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35129.39 MB 2025-02-14 07:41:38,085 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 07:41:38,085 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32383.95 MB 2025-02-14 07:41:38,086 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 07:41:38,086 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 07:41:38,086 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 07:41:38,086 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:41:38,086 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22708.28 MB 2025-02-14 07:41:38,086 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26839.67 MB 2025-02-14 07:41:38,086 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 07:41:38,086 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29467.08 MB 2025-02-14 07:41:38,086 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35129.39 MB 2025-02-14 07:41:38,086 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 07:41:38,086 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32383.95 MB 2025-02-14 07:41:38,256 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 07:41:38,256 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 07:41:38,256 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 07:41:38,256 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:41:38,256 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28373.21 MB 2025-02-14 07:41:38,256 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29140.21 MB 2025-02-14 07:41:38,256 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 07:41:38,256 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35129.39 MB 2025-02-14 07:41:38,256 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35544.63 MB 2025-02-14 07:41:38,256 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 07:41:38,256 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29848.00 MB 2025-02-14 07:41:38,275 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 07:41:38,275 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 07:41:38,275 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:41:38,275 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:41:38,275 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29553.10 MB 2025-02-14 07:41:38,275 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29781.94 MB 2025-02-14 07:41:38,275 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.84 MB 2025-02-14 07:41:38,275 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35544.63 MB 2025-02-14 07:41:38,275 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35544.63 MB 2025-02-14 07:41:38,275 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:41:38,275 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30017.66 MB 2025-02-14 07:41:38,276 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 07:41:38,277 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 07:41:38,277 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.46 seconds 2025-02-14 07:41:38,277 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:41:38,277 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17257.61 MB 2025-02-14 07:41:38,277 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29982.49 MB 2025-02-14 07:41:38,277 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12724.89 MB 2025-02-14 07:41:38,277 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39799.75 MB 2025-02-14 07:41:38,277 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35544.63 MB 2025-02-14 07:41:38,277 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4255.12 MB 2025-02-14 07:41:38,277 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30017.66 MB 2025-02-14 07:41:38,545 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 07:41:38,545 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 07:41:38,545 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 07:41:38,545 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:41:38,545 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29982.49 MB 2025-02-14 07:41:38,545 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22254.00 MB 2025-02-14 07:41:38,545 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7728.50 MB 2025-02-14 07:41:38,545 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35544.63 MB 2025-02-14 07:41:38,545 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35544.63 MB 2025-02-14 07:41:38,545 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:41:38,545 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32487.71 MB 2025-02-14 07:41:38,563 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8141, cut from 8143 2025-02-14 07:41:38,563 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2 ('] 2025-02-14 07:41:38,569 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 07:41:38,569 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 07:41:38,569 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:41:38,569 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:41:38,569 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22254.00 MB 2025-02-14 07:41:38,569 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30671.12 MB 2025-02-14 07:41:38,569 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8417.12 MB 2025-02-14 07:41:38,569 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35544.63 MB 2025-02-14 07:41:38,569 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39728.45 MB 2025-02-14 07:41:38,569 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4183.82 MB 2025-02-14 07:41:38,569 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30671.12 MB 2025-02-14 07:41:38,731 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7933] 2025-02-14 07:41:38,732 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:41:38,732 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 07:41:38,733 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:41:38,733 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 07:41:38,738 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 07:41:38,739 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:41:38,739 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 07:41:38,739 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2 ('] 2025-02-14 07:41:47,574 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:41:47,574 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 07:41:47,579 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 07:41:47,582 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:41:47,582 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 144, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 07:41:47,583 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:41:47,583 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 144, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 07:41:49,876 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 07:41:49,877 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 07:41:49,877 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.29 seconds 2025-02-14 07:41:49,877 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:41:49,877 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13972.12 MB 2025-02-14 07:41:49,877 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14481.73 MB 2025-02-14 07:41:49,877 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 509.61 MB 2025-02-14 07:41:49,877 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48096.08 MB 2025-02-14 07:41:49,877 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17383.29 MB 2025-02-14 07:41:49,877 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30712.79 MB 2025-02-14 07:41:49,877 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23444.30 MB 2025-02-14 07:41:49,888 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 07:41:49,888 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 07:41:49,888 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:41:49,888 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:41:49,888 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14481.73 MB 2025-02-14 07:41:49,888 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14728.63 MB 2025-02-14 07:41:49,888 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 246.90 MB 2025-02-14 07:41:49,888 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17383.29 MB 2025-02-14 07:41:49,888 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17892.90 MB 2025-02-14 07:41:49,888 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 509.61 MB 2025-02-14 07:41:49,888 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16561.84 MB 2025-02-14 07:41:50,589 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 07:41:50,589 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 07:41:50,590 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.70 seconds 2025-02-14 07:41:50,590 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:41:50,590 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14728.63 MB 2025-02-14 07:41:50,590 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14919.74 MB 2025-02-14 07:41:50,590 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 191.10 MB 2025-02-14 07:41:50,590 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17892.90 MB 2025-02-14 07:41:50,590 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17892.90 MB 2025-02-14 07:41:50,590 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:41:50,590 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18900.11 MB 2025-02-14 07:41:50,598 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 07:41:50,598 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 07:41:50,598 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 07:41:50,598 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:41:50,598 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14919.67 MB 2025-02-14 07:41:50,598 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15599.74 MB 2025-02-14 07:41:50,598 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 680.07 MB 2025-02-14 07:41:50,598 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17892.90 MB 2025-02-14 07:41:50,598 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17892.90 MB 2025-02-14 07:41:50,598 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:41:50,598 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16110.02 MB 2025-02-14 07:41:50,679 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 07:41:50,680 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 07:41:50,680 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 07:41:50,680 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:41:50,680 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15599.74 MB 2025-02-14 07:41:50,680 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16406.85 MB 2025-02-14 07:41:50,680 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 807.11 MB 2025-02-14 07:41:50,680 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17892.90 MB 2025-02-14 07:41:50,680 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19251.86 MB 2025-02-14 07:41:50,680 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1358.95 MB 2025-02-14 07:41:50,680 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18402.75 MB 2025-02-14 07:41:50,680 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 07:41:50,680 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 07:41:50,680 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 07:41:50,680 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:41:50,680 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14919.67 MB 2025-02-14 07:41:50,680 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16406.85 MB 2025-02-14 07:41:50,680 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1487.18 MB 2025-02-14 07:41:50,680 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17892.90 MB 2025-02-14 07:41:50,680 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19251.86 MB 2025-02-14 07:41:50,680 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1358.95 MB 2025-02-14 07:41:50,680 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18402.75 MB 2025-02-14 07:41:50,742 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 07:41:50,742 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 07:41:50,742 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 07:41:50,742 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:41:50,742 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16958.92 MB 2025-02-14 07:41:50,742 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17235.04 MB 2025-02-14 07:41:50,742 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 276.12 MB 2025-02-14 07:41:50,742 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19251.86 MB 2025-02-14 07:41:50,742 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19396.56 MB 2025-02-14 07:41:50,742 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 144.70 MB 2025-02-14 07:41:50,742 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17500.12 MB 2025-02-14 07:41:50,750 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 07:41:50,750 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 07:41:50,750 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:41:50,750 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:41:50,750 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17383.69 MB 2025-02-14 07:41:50,750 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17611.67 MB 2025-02-14 07:41:50,750 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.98 MB 2025-02-14 07:41:50,750 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19396.56 MB 2025-02-14 07:41:50,750 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19396.56 MB 2025-02-14 07:41:50,750 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:41:50,750 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17613.56 MB 2025-02-14 07:41:50,751 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 07:41:50,752 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 07:41:50,752 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.17 seconds 2025-02-14 07:41:50,752 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:41:50,752 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13470.41 MB 2025-02-14 07:41:50,752 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17812.67 MB 2025-02-14 07:41:50,752 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4342.26 MB 2025-02-14 07:41:50,752 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48096.08 MB 2025-02-14 07:41:50,752 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19396.56 MB 2025-02-14 07:41:50,752 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28699.53 MB 2025-02-14 07:41:50,752 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17812.67 MB 2025-02-14 07:41:51,020 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 07:41:51,020 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 07:41:51,020 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 07:41:51,020 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:41:51,020 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17812.67 MB 2025-02-14 07:41:51,020 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17265.67 MB 2025-02-14 07:41:51,020 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -547.00 MB 2025-02-14 07:41:51,020 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19396.56 MB 2025-02-14 07:41:51,020 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19664.99 MB 2025-02-14 07:41:51,020 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 268.44 MB 2025-02-14 07:41:51,020 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19018.14 MB 2025-02-14 07:41:51,038 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8159, cut from 8161 2025-02-14 07:41:51,038 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 07:41:51,044 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 07:41:51,045 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 07:41:51,045 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:41:51,045 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:41:51,045 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17265.67 MB 2025-02-14 07:41:51,045 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25701.26 MB 2025-02-14 07:41:51,045 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8435.59 MB 2025-02-14 07:41:51,045 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19664.99 MB 2025-02-14 07:41:51,045 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30150.75 MB 2025-02-14 07:41:51,045 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10485.76 MB 2025-02-14 07:41:51,045 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25701.26 MB 2025-02-14 07:41:51,205 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7951] 2025-02-14 07:41:51,207 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:41:51,207 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 07:41:51,208 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:41:51,208 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 07:41:51,212 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 07:41:51,214 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:41:51,214 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 07:41:51,214 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 07:42:01,740 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:42:01,741 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 07:42:01,746 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 07:42:01,749 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:42:01,749 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 173, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 07:42:01,750 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:42:01,750 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 173, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 07:42:04,443 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 07:42:04,443 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 07:42:04,443 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.69 seconds 2025-02-14 07:42:04,443 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:42:04,443 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14174.20 MB 2025-02-14 07:42:04,443 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14786.44 MB 2025-02-14 07:42:04,443 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 612.24 MB 2025-02-14 07:42:04,443 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38539.36 MB 2025-02-14 07:42:04,443 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18270.39 MB 2025-02-14 07:42:04,443 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20268.97 MB 2025-02-14 07:42:04,443 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23646.38 MB 2025-02-14 07:42:04,456 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 07:42:04,456 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 07:42:04,456 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:42:04,456 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:42:04,456 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14786.44 MB 2025-02-14 07:42:04,456 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15083.06 MB 2025-02-14 07:42:04,456 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 296.63 MB 2025-02-14 07:42:04,456 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18270.39 MB 2025-02-14 07:42:04,456 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18270.39 MB 2025-02-14 07:42:04,456 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:42:04,456 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17262.47 MB 2025-02-14 07:42:05,293 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 07:42:05,293 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 07:42:05,293 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.83 seconds 2025-02-14 07:42:05,293 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:42:05,293 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15083.06 MB 2025-02-14 07:42:05,293 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15312.65 MB 2025-02-14 07:42:05,293 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.59 MB 2025-02-14 07:42:05,293 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18270.39 MB 2025-02-14 07:42:05,293 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18213.77 MB 2025-02-14 07:42:05,293 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -56.62 MB 2025-02-14 07:42:05,293 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19254.54 MB 2025-02-14 07:42:05,301 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 07:42:05,301 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 07:42:05,301 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:42:05,301 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:42:05,301 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15312.59 MB 2025-02-14 07:42:05,301 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16129.61 MB 2025-02-14 07:42:05,301 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 817.03 MB 2025-02-14 07:42:05,301 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18213.77 MB 2025-02-14 07:42:05,301 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18213.77 MB 2025-02-14 07:42:05,301 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:42:05,301 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16742.65 MB 2025-02-14 07:42:05,397 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 07:42:05,397 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 07:42:05,397 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 07:42:05,397 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:42:05,397 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16129.61 MB 2025-02-14 07:42:05,397 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17100.04 MB 2025-02-14 07:42:05,397 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 970.43 MB 2025-02-14 07:42:05,397 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18213.77 MB 2025-02-14 07:42:05,397 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20667.43 MB 2025-02-14 07:42:05,397 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2453.67 MB 2025-02-14 07:42:05,397 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19501.05 MB 2025-02-14 07:42:05,398 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 07:42:05,398 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 07:42:05,398 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 07:42:05,398 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:42:05,398 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15312.59 MB 2025-02-14 07:42:05,398 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17100.04 MB 2025-02-14 07:42:05,398 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1787.45 MB 2025-02-14 07:42:05,398 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18213.77 MB 2025-02-14 07:42:05,398 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20667.43 MB 2025-02-14 07:42:05,398 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2453.67 MB 2025-02-14 07:42:05,398 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19501.05 MB 2025-02-14 07:42:05,473 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 07:42:05,473 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 07:42:05,473 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 07:42:05,473 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:42:05,473 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17763.29 MB 2025-02-14 07:42:05,473 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18095.81 MB 2025-02-14 07:42:05,473 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 332.52 MB 2025-02-14 07:42:05,473 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20667.43 MB 2025-02-14 07:42:05,474 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20841.50 MB 2025-02-14 07:42:05,474 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 174.06 MB 2025-02-14 07:42:05,474 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18409.80 MB 2025-02-14 07:42:05,483 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 07:42:05,484 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 07:42:05,484 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:42:05,484 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:42:05,484 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18274.39 MB 2025-02-14 07:42:05,484 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18488.13 MB 2025-02-14 07:42:05,484 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 213.74 MB 2025-02-14 07:42:05,484 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20841.50 MB 2025-02-14 07:42:05,484 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20843.59 MB 2025-02-14 07:42:05,484 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-14 07:42:05,484 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18513.59 MB 2025-02-14 07:42:05,485 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 07:42:05,485 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 07:42:05,485 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.73 seconds 2025-02-14 07:42:05,485 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:42:05,485 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13571.45 MB 2025-02-14 07:42:05,485 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18689.20 MB 2025-02-14 07:42:05,485 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5117.75 MB 2025-02-14 07:42:05,485 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38539.36 MB 2025-02-14 07:42:05,485 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20843.59 MB 2025-02-14 07:42:05,485 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17695.77 MB 2025-02-14 07:42:05,485 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18689.20 MB 2025-02-14 07:42:05,755 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 07:42:05,755 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 07:42:05,755 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 07:42:05,755 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:42:05,755 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18689.20 MB 2025-02-14 07:42:05,755 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17505.35 MB 2025-02-14 07:42:05,755 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1183.85 MB 2025-02-14 07:42:05,755 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20843.59 MB 2025-02-14 07:42:05,755 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20843.59 MB 2025-02-14 07:42:05,755 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:42:05,755 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18923.62 MB 2025-02-14 07:42:05,773 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 07:42:05,773 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 07:42:05,779 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 07:42:05,779 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 07:42:05,779 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:42:05,779 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:42:05,779 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17505.35 MB 2025-02-14 07:42:05,779 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25944.37 MB 2025-02-14 07:42:05,779 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 07:42:05,779 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20843.59 MB 2025-02-14 07:42:05,779 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31333.55 MB 2025-02-14 07:42:05,779 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 07:42:05,779 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25944.37 MB 2025-02-14 07:42:05,948 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 07:42:05,949 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:42:05,949 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 07:42:05,950 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:42:05,950 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 07:42:05,955 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 07:42:05,956 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:42:05,956 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 07:42:05,957 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 07:43:08,702 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:43:08,703 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 07:43:08,707 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 07:43:08,711 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:43:08,711 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 193, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 07:43:08,712 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:43:08,712 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 193, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 07:43:11,710 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 07:43:11,710 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 07:43:11,710 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.00 seconds 2025-02-14 07:43:11,711 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:43:11,711 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14313.56 MB 2025-02-14 07:43:11,711 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14996.58 MB 2025-02-14 07:43:11,711 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 683.02 MB 2025-02-14 07:43:11,711 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43918.56 MB 2025-02-14 07:43:11,711 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17997.76 MB 2025-02-14 07:43:11,711 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25920.80 MB 2025-02-14 07:43:11,711 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24012.23 MB 2025-02-14 07:43:11,725 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 07:43:11,725 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 07:43:11,725 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:43:11,725 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:43:11,725 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14996.58 MB 2025-02-14 07:43:11,725 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15327.50 MB 2025-02-14 07:43:11,725 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 330.92 MB 2025-02-14 07:43:11,725 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17997.76 MB 2025-02-14 07:43:11,725 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18681.43 MB 2025-02-14 07:43:11,725 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 683.67 MB 2025-02-14 07:43:11,725 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17711.72 MB 2025-02-14 07:43:12,658 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 07:43:12,659 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 07:43:12,659 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.93 seconds 2025-02-14 07:43:12,659 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:43:12,659 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15327.50 MB 2025-02-14 07:43:12,659 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15583.63 MB 2025-02-14 07:43:12,659 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 256.13 MB 2025-02-14 07:43:12,659 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18681.43 MB 2025-02-14 07:43:12,659 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17588.81 MB 2025-02-14 07:43:12,659 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1092.62 MB 2025-02-14 07:43:12,659 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19583.91 MB 2025-02-14 07:43:12,667 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 07:43:12,667 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 07:43:12,667 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:43:12,667 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:43:12,667 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15583.56 MB 2025-02-14 07:43:12,667 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16495.04 MB 2025-02-14 07:43:12,667 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 911.48 MB 2025-02-14 07:43:12,667 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17588.81 MB 2025-02-14 07:43:12,667 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18503.17 MB 2025-02-14 07:43:12,667 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 914.36 MB 2025-02-14 07:43:12,667 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17178.96 MB 2025-02-14 07:43:12,773 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 07:43:12,773 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 07:43:12,773 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 07:43:12,773 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:43:12,773 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16495.04 MB 2025-02-14 07:43:12,773 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17577.26 MB 2025-02-14 07:43:12,773 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1082.22 MB 2025-02-14 07:43:12,773 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18503.17 MB 2025-02-14 07:43:12,773 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21246.25 MB 2025-02-14 07:43:12,773 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2743.07 MB 2025-02-14 07:43:12,773 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20254.18 MB 2025-02-14 07:43:12,774 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 07:43:12,774 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 07:43:12,774 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 07:43:12,774 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:43:12,774 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15583.56 MB 2025-02-14 07:43:12,774 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17577.26 MB 2025-02-14 07:43:12,774 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1993.70 MB 2025-02-14 07:43:12,774 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17588.81 MB 2025-02-14 07:43:12,774 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21246.25 MB 2025-02-14 07:43:12,774 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3657.43 MB 2025-02-14 07:43:12,774 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20254.18 MB 2025-02-14 07:43:12,856 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 07:43:12,856 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 07:43:12,856 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 07:43:12,856 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:43:12,856 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18317.20 MB 2025-02-14 07:43:12,856 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18688.19 MB 2025-02-14 07:43:12,856 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 371.00 MB 2025-02-14 07:43:12,856 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21246.25 MB 2025-02-14 07:43:12,856 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21443.38 MB 2025-02-14 07:43:12,856 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 197.13 MB 2025-02-14 07:43:12,856 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19035.00 MB 2025-02-14 07:43:12,867 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 07:43:12,867 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 07:43:12,867 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:43:12,867 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:43:12,867 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18887.42 MB 2025-02-14 07:43:12,867 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19117.33 MB 2025-02-14 07:43:12,867 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.91 MB 2025-02-14 07:43:12,867 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21443.38 MB 2025-02-14 07:43:12,867 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21443.38 MB 2025-02-14 07:43:12,867 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:43:12,867 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19162.76 MB 2025-02-14 07:43:12,868 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 07:43:12,868 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 07:43:12,868 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.15 seconds 2025-02-14 07:43:12,868 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:43:12,868 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13641.13 MB 2025-02-14 07:43:12,868 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19318.40 MB 2025-02-14 07:43:12,868 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5677.27 MB 2025-02-14 07:43:12,868 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43918.56 MB 2025-02-14 07:43:12,868 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21443.38 MB 2025-02-14 07:43:12,868 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22475.18 MB 2025-02-14 07:43:12,868 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19318.40 MB 2025-02-14 07:43:13,135 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 07:43:13,135 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 07:43:13,135 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 07:43:13,135 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:43:13,135 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19318.40 MB 2025-02-14 07:43:13,135 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17669.28 MB 2025-02-14 07:43:13,135 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1649.12 MB 2025-02-14 07:43:13,135 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21443.38 MB 2025-02-14 07:43:13,135 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21443.38 MB 2025-02-14 07:43:13,135 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:43:13,135 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19318.40 MB 2025-02-14 07:43:13,153 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 07:43:13,153 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2,'] 2025-02-14 07:43:13,159 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 07:43:13,159 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 07:43:13,159 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:43:13,159 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:43:13,159 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17669.28 MB 2025-02-14 07:43:13,159 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26108.31 MB 2025-02-14 07:43:13,159 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 07:43:13,159 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21443.38 MB 2025-02-14 07:43:13,159 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31933.33 MB 2025-02-14 07:43:13,159 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 07:43:13,159 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26108.31 MB 2025-02-14 07:43:13,322 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 07:43:13,324 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:43:13,324 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 07:43:13,325 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:43:13,325 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 07:43:13,329 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 07:43:13,330 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:43:13,330 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 07:43:13,331 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2,'] 2025-02-14 07:43:38,878 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:43:38,878 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 07:43:38,883 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 07:43:38,886 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:43:38,887 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1107, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 07:43:38,887 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:43:38,887 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1107, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 07:43:55,959 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 07:43:55,959 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 07:43:55,959 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.07 seconds 2025-02-14 07:43:55,959 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:43:55,959 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20682.46 MB 2025-02-14 07:43:55,959 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24600.07 MB 2025-02-14 07:43:55,959 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3917.61 MB 2025-02-14 07:43:55,959 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44518.34 MB 2025-02-14 07:43:55,959 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31314.67 MB 2025-02-14 07:43:55,960 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13203.67 MB 2025-02-14 07:43:55,960 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33551.22 MB 2025-02-14 07:43:56,028 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 07:43:56,028 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 07:43:56,028 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 07:43:56,028 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:43:56,028 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24600.07 MB 2025-02-14 07:43:56,028 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21532.80 MB 2025-02-14 07:43:56,028 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3067.27 MB 2025-02-14 07:43:56,028 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31314.67 MB 2025-02-14 07:43:56,028 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40944.80 MB 2025-02-14 07:43:56,028 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9630.12 MB 2025-02-14 07:43:56,028 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36153.62 MB 2025-02-14 07:43:57,942 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 07:43:57,942 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 07:43:57,942 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 07:43:57,942 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:43:57,942 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21532.80 MB 2025-02-14 07:43:57,942 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22063.64 MB 2025-02-14 07:43:57,942 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 07:43:57,942 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40944.80 MB 2025-02-14 07:43:57,942 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28810.67 MB 2025-02-14 07:43:57,942 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12134.12 MB 2025-02-14 07:43:57,942 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26042.97 MB 2025-02-14 07:43:57,955 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 07:43:57,955 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 07:43:57,955 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:43:57,955 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:43:57,955 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22063.64 MB 2025-02-14 07:43:57,955 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23953.17 MB 2025-02-14 07:43:57,955 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 07:43:57,955 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28810.67 MB 2025-02-14 07:43:57,955 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28810.67 MB 2025-02-14 07:43:57,955 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:43:57,955 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25370.60 MB 2025-02-14 07:43:58,168 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 07:43:58,168 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 07:43:58,168 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 07:43:58,168 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:43:58,168 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23953.17 MB 2025-02-14 07:43:58,168 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26195.03 MB 2025-02-14 07:43:58,168 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 07:43:58,168 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28810.67 MB 2025-02-14 07:43:58,168 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34472.98 MB 2025-02-14 07:43:58,168 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 07:43:58,168 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31739.31 MB 2025-02-14 07:43:58,169 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 07:43:58,169 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 07:43:58,169 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 07:43:58,169 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:43:58,169 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22063.64 MB 2025-02-14 07:43:58,169 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26195.03 MB 2025-02-14 07:43:58,169 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 07:43:58,169 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28810.67 MB 2025-02-14 07:43:58,169 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34472.98 MB 2025-02-14 07:43:58,169 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 07:43:58,169 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31739.31 MB 2025-02-14 07:43:58,341 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 07:43:58,341 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 07:43:58,341 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 07:43:58,341 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:43:58,341 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27728.57 MB 2025-02-14 07:43:58,341 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28495.57 MB 2025-02-14 07:43:58,341 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 07:43:58,341 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34472.98 MB 2025-02-14 07:43:58,341 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34890.32 MB 2025-02-14 07:43:58,341 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 07:43:58,341 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29203.36 MB 2025-02-14 07:43:58,360 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 07:43:58,360 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 07:43:58,360 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:43:58,360 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:43:58,360 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28908.46 MB 2025-02-14 07:43:58,360 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29135.19 MB 2025-02-14 07:43:58,360 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.72 MB 2025-02-14 07:43:58,360 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34890.32 MB 2025-02-14 07:43:58,360 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34890.32 MB 2025-02-14 07:43:58,360 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:43:58,360 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29351.62 MB 2025-02-14 07:43:58,361 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 07:43:58,361 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 07:43:58,361 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.47 seconds 2025-02-14 07:43:58,361 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:43:58,361 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16825.58 MB 2025-02-14 07:43:58,361 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29335.99 MB 2025-02-14 07:43:58,361 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12510.41 MB 2025-02-14 07:43:58,361 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44518.34 MB 2025-02-14 07:43:58,362 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34890.32 MB 2025-02-14 07:43:58,362 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9628.02 MB 2025-02-14 07:43:58,362 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29351.62 MB 2025-02-14 07:43:58,630 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 07:43:58,630 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 07:43:58,630 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 07:43:58,630 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:43:58,630 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29335.99 MB 2025-02-14 07:43:58,630 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21825.78 MB 2025-02-14 07:43:58,630 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7510.21 MB 2025-02-14 07:43:58,630 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34890.32 MB 2025-02-14 07:43:58,630 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34890.32 MB 2025-02-14 07:43:58,630 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:43:58,630 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31844.28 MB 2025-02-14 07:43:58,648 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8151, cut from 8153 2025-02-14 07:43:58,648 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-14 07:43:58,654 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 07:43:58,654 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 07:43:58,654 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:43:58,654 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:43:58,654 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21825.78 MB 2025-02-14 07:43:58,654 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30253.12 MB 2025-02-14 07:43:58,654 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8427.34 MB 2025-02-14 07:43:58,654 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34890.32 MB 2025-02-14 07:43:58,654 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43270.54 MB 2025-02-14 07:43:58,654 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8380.22 MB 2025-02-14 07:43:58,654 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30253.12 MB 2025-02-14 07:43:58,817 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7943] 2025-02-14 07:43:58,818 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:43:58,818 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 07:43:58,819 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:43:58,819 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 07:43:58,824 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 07:43:58,825 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:43:58,825 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 07:43:58,825 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-14 07:45:34,775 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:45:34,776 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 07:45:34,781 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 07:45:34,784 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:45:34,784 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 492, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 07:45:34,785 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:45:34,785 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 492, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 07:45:42,332 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 07:45:42,332 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 07:45:42,332 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.54 seconds 2025-02-14 07:45:42,332 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:45:42,332 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16397.04 MB 2025-02-14 07:45:42,332 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18138.20 MB 2025-02-14 07:45:42,332 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1741.16 MB 2025-02-14 07:45:42,332 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51650.76 MB 2025-02-14 07:45:42,332 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22426.94 MB 2025-02-14 07:45:42,332 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29223.81 MB 2025-02-14 07:45:42,332 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27001.68 MB 2025-02-14 07:45:42,365 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 07:45:42,365 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 07:45:42,366 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 07:45:42,366 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:45:42,366 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18138.20 MB 2025-02-14 07:45:42,366 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18335.61 MB 2025-02-14 07:45:42,366 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 197.40 MB 2025-02-14 07:45:42,366 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22426.94 MB 2025-02-14 07:45:42,366 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28554.82 MB 2025-02-14 07:45:42,366 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6127.88 MB 2025-02-14 07:45:42,366 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25699.84 MB 2025-02-14 07:45:44,273 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 07:45:44,273 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 07:45:44,273 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 07:45:44,273 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:45:44,273 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18335.61 MB 2025-02-14 07:45:44,273 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18866.45 MB 2025-02-14 07:45:44,273 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 07:45:44,273 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28554.82 MB 2025-02-14 07:45:44,273 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21156.07 MB 2025-02-14 07:45:44,273 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7398.75 MB 2025-02-14 07:45:44,273 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22846.82 MB 2025-02-14 07:45:44,287 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 07:45:44,287 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 07:45:44,287 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:45:44,287 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:45:44,287 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18866.45 MB 2025-02-14 07:45:44,287 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20755.98 MB 2025-02-14 07:45:44,287 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 07:45:44,287 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21156.07 MB 2025-02-14 07:45:44,287 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24459.08 MB 2025-02-14 07:45:44,287 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 07:45:44,287 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22173.41 MB 2025-02-14 07:45:44,494 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 07:45:44,494 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 07:45:44,494 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 07:45:44,494 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:45:44,494 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20755.98 MB 2025-02-14 07:45:44,494 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22997.84 MB 2025-02-14 07:45:44,494 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 07:45:44,494 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24459.08 MB 2025-02-14 07:45:44,494 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31065.11 MB 2025-02-14 07:45:44,494 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 07:45:44,494 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28542.12 MB 2025-02-14 07:45:44,494 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 07:45:44,494 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 07:45:44,494 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 07:45:44,495 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:45:44,495 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18866.45 MB 2025-02-14 07:45:44,495 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22997.84 MB 2025-02-14 07:45:44,495 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 07:45:44,495 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21156.07 MB 2025-02-14 07:45:44,495 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31065.11 MB 2025-02-14 07:45:44,495 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9909.04 MB 2025-02-14 07:45:44,495 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28542.12 MB 2025-02-14 07:45:44,653 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 07:45:44,653 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 07:45:44,653 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 07:45:44,653 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:45:44,653 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24531.38 MB 2025-02-14 07:45:44,653 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25298.38 MB 2025-02-14 07:45:44,653 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 07:45:44,653 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31065.11 MB 2025-02-14 07:45:44,653 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31480.35 MB 2025-02-14 07:45:44,653 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 07:45:44,653 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26006.17 MB 2025-02-14 07:45:44,671 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 07:45:44,671 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 07:45:44,671 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:45:44,672 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:45:44,672 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25711.27 MB 2025-02-14 07:45:44,672 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25940.28 MB 2025-02-14 07:45:44,672 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.01 MB 2025-02-14 07:45:44,672 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31480.35 MB 2025-02-14 07:45:44,672 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31480.35 MB 2025-02-14 07:45:44,672 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:45:44,672 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26132.02 MB 2025-02-14 07:45:44,673 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 07:45:44,673 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 07:45:44,673 - resource_logging.py:150 - __exit__ - DEBUG - Time: 9.89 seconds 2025-02-14 07:45:44,673 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:45:44,673 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14682.87 MB 2025-02-14 07:45:44,673 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26141.35 MB 2025-02-14 07:45:44,673 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11458.48 MB 2025-02-14 07:45:44,673 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51650.76 MB 2025-02-14 07:45:44,673 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31480.35 MB 2025-02-14 07:45:44,673 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20170.41 MB 2025-02-14 07:45:44,673 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26141.35 MB 2025-02-14 07:45:44,937 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 07:45:44,937 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 07:45:44,937 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 07:45:44,937 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:45:44,937 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26141.35 MB 2025-02-14 07:45:44,937 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19687.26 MB 2025-02-14 07:45:44,937 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6454.09 MB 2025-02-14 07:45:44,937 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31480.35 MB 2025-02-14 07:45:44,937 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31480.35 MB 2025-02-14 07:45:44,937 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:45:44,937 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28653.02 MB 2025-02-14 07:45:44,955 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 07:45:44,956 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 07:45:44,962 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 07:45:44,962 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 07:45:44,962 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:45:44,962 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:45:44,962 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19687.26 MB 2025-02-14 07:45:44,962 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28126.29 MB 2025-02-14 07:45:44,962 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 07:45:44,962 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31480.35 MB 2025-02-14 07:45:44,962 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41970.30 MB 2025-02-14 07:45:44,962 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 07:45:44,962 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28126.29 MB 2025-02-14 07:45:45,114 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 07:45:45,116 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:45:45,116 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 07:45:45,117 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:45:45,117 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 07:45:45,121 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 07:45:45,122 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:45:45,122 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 07:45:45,122 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 07:46:37,086 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:46:37,086 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 07:46:37,091 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 07:46:37,095 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:46:37,095 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2154, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 07:46:37,096 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:46:37,096 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2154, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 07:47:10,332 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 07:47:10,332 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 07:47:10,332 - resource_logging.py:150 - __exit__ - DEBUG - Time: 33.23 seconds 2025-02-14 07:47:10,332 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:47:10,332 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27978.12 MB 2025-02-14 07:47:10,332 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35601.27 MB 2025-02-14 07:47:10,332 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7623.15 MB 2025-02-14 07:47:10,332 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54555.31 MB 2025-02-14 07:47:10,332 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41313.89 MB 2025-02-14 07:47:10,332 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13241.42 MB 2025-02-14 07:47:10,332 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44471.57 MB 2025-02-14 07:47:10,468 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 07:47:10,468 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 07:47:10,468 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 07:47:10,468 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:47:10,468 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35601.27 MB 2025-02-14 07:47:10,468 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26975.82 MB 2025-02-14 07:47:10,468 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8625.45 MB 2025-02-14 07:47:10,468 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41313.89 MB 2025-02-14 07:47:10,468 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 67526.20 MB 2025-02-14 07:47:10,468 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 26212.30 MB 2025-02-14 07:47:10,468 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 57219.32 MB 2025-02-14 07:47:12,428 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 07:47:12,428 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 07:47:12,428 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.96 seconds 2025-02-14 07:47:12,428 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:47:12,428 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26975.82 MB 2025-02-14 07:47:12,428 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27506.67 MB 2025-02-14 07:47:12,428 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 07:47:12,428 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 67526.20 MB 2025-02-14 07:47:12,428 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30912.02 MB 2025-02-14 07:47:12,428 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -36614.18 MB 2025-02-14 07:47:12,428 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31487.04 MB 2025-02-14 07:47:12,442 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 07:47:12,442 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 07:47:12,442 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:47:12,442 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:47:12,442 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27506.67 MB 2025-02-14 07:47:12,442 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29396.20 MB 2025-02-14 07:47:12,442 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 07:47:12,442 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30912.02 MB 2025-02-14 07:47:12,442 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33743.18 MB 2025-02-14 07:47:12,442 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-14 07:47:12,442 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30813.63 MB 2025-02-14 07:47:12,650 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 07:47:12,650 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 07:47:12,650 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 07:47:12,650 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:47:12,650 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29396.20 MB 2025-02-14 07:47:12,650 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31638.06 MB 2025-02-14 07:47:12,650 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 07:47:12,650 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33743.18 MB 2025-02-14 07:47:12,650 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39405.49 MB 2025-02-14 07:47:12,650 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 07:47:12,650 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37182.34 MB 2025-02-14 07:47:12,650 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 07:47:12,650 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 07:47:12,650 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 07:47:12,650 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:47:12,651 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27506.67 MB 2025-02-14 07:47:12,651 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31638.06 MB 2025-02-14 07:47:12,651 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 07:47:12,651 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30912.02 MB 2025-02-14 07:47:12,651 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39405.49 MB 2025-02-14 07:47:12,651 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8493.47 MB 2025-02-14 07:47:12,651 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37182.34 MB 2025-02-14 07:47:12,817 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 07:47:12,817 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 07:47:12,817 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 07:47:12,817 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:47:12,817 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33171.60 MB 2025-02-14 07:47:12,817 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33938.60 MB 2025-02-14 07:47:12,817 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 07:47:12,817 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39405.49 MB 2025-02-14 07:47:12,817 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39818.63 MB 2025-02-14 07:47:12,817 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 07:47:12,817 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34646.39 MB 2025-02-14 07:47:12,836 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 07:47:12,836 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 07:47:12,836 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:47:12,836 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:47:12,836 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34351.49 MB 2025-02-14 07:47:12,836 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34580.11 MB 2025-02-14 07:47:12,836 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.62 MB 2025-02-14 07:47:12,836 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39818.63 MB 2025-02-14 07:47:12,837 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39818.63 MB 2025-02-14 07:47:12,837 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:47:12,837 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34809.59 MB 2025-02-14 07:47:12,838 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 07:47:12,838 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 07:47:12,838 - resource_logging.py:150 - __exit__ - DEBUG - Time: 35.74 seconds 2025-02-14 07:47:12,838 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:47:12,838 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20473.41 MB 2025-02-14 07:47:12,838 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34780.64 MB 2025-02-14 07:47:12,838 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14307.22 MB 2025-02-14 07:47:12,838 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54555.31 MB 2025-02-14 07:47:12,838 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39818.63 MB 2025-02-14 07:47:12,838 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14736.69 MB 2025-02-14 07:47:12,838 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34809.59 MB 2025-02-14 07:47:13,107 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 07:47:13,107 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 07:47:13,107 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 07:47:13,107 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:47:13,107 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34780.64 MB 2025-02-14 07:47:13,107 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25469.42 MB 2025-02-14 07:47:13,107 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9311.21 MB 2025-02-14 07:47:13,107 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39818.63 MB 2025-02-14 07:47:13,107 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39818.63 MB 2025-02-14 07:47:13,107 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:47:13,107 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37285.55 MB 2025-02-14 07:47:13,125 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8140, cut from 8142 2025-02-14 07:47:13,125 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 07:47:13,131 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 07:47:13,131 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 07:47:13,131 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:47:13,131 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:47:13,131 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25469.42 MB 2025-02-14 07:47:13,131 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33886.03 MB 2025-02-14 07:47:13,131 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8416.60 MB 2025-02-14 07:47:13,131 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39818.63 MB 2025-02-14 07:47:13,131 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48186.26 MB 2025-02-14 07:47:13,131 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8367.64 MB 2025-02-14 07:47:13,132 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33886.03 MB 2025-02-14 07:47:13,297 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7932] 2025-02-14 07:47:13,299 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:47:13,299 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 07:47:13,300 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:47:13,300 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 07:47:13,305 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 07:47:13,306 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:47:13,306 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 07:47:13,306 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 07:47:58,257 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:47:58,257 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 07:47:58,263 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 07:47:58,269 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:47:58,269 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1183, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 07:47:58,271 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:47:58,271 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1183, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 07:48:16,667 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 07:48:16,667 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 07:48:16,667 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.39 seconds 2025-02-14 07:48:16,667 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:48:16,667 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21212.04 MB 2025-02-14 07:48:16,667 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25398.61 MB 2025-02-14 07:48:16,667 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4186.57 MB 2025-02-14 07:48:16,667 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56553.90 MB 2025-02-14 07:48:16,667 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29479.67 MB 2025-02-14 07:48:16,667 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27074.23 MB 2025-02-14 07:48:16,667 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34308.10 MB 2025-02-14 07:48:16,749 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 07:48:16,749 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 07:48:16,749 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 07:48:16,749 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:48:16,749 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25398.61 MB 2025-02-14 07:48:16,749 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21927.90 MB 2025-02-14 07:48:16,749 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3470.71 MB 2025-02-14 07:48:16,749 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29479.67 MB 2025-02-14 07:48:16,749 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45231.37 MB 2025-02-14 07:48:16,749 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 15751.71 MB 2025-02-14 07:48:16,749 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37984.72 MB 2025-02-14 07:48:18,677 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 07:48:18,678 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 07:48:18,678 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 07:48:18,678 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:48:18,678 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21927.90 MB 2025-02-14 07:48:18,678 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22458.74 MB 2025-02-14 07:48:18,678 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 07:48:18,678 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45231.37 MB 2025-02-14 07:48:18,678 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26707.23 MB 2025-02-14 07:48:18,678 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18524.14 MB 2025-02-14 07:48:18,678 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26438.07 MB 2025-02-14 07:48:18,691 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 07:48:18,691 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 07:48:18,691 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:48:18,691 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:48:18,691 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22458.74 MB 2025-02-14 07:48:18,691 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24348.27 MB 2025-02-14 07:48:18,691 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 07:48:18,691 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26707.23 MB 2025-02-14 07:48:18,691 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28594.67 MB 2025-02-14 07:48:18,691 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 07:48:18,691 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25765.70 MB 2025-02-14 07:48:18,904 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 07:48:18,904 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 07:48:18,904 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 07:48:18,904 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:48:18,904 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24348.27 MB 2025-02-14 07:48:18,904 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26590.13 MB 2025-02-14 07:48:18,904 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 07:48:18,904 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28594.67 MB 2025-02-14 07:48:18,904 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34256.98 MB 2025-02-14 07:48:18,904 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 07:48:18,904 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32134.41 MB 2025-02-14 07:48:18,905 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 07:48:18,905 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 07:48:18,905 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 07:48:18,905 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:48:18,905 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22458.74 MB 2025-02-14 07:48:18,905 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26590.13 MB 2025-02-14 07:48:18,905 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 07:48:18,905 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26707.23 MB 2025-02-14 07:48:18,905 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34256.98 MB 2025-02-14 07:48:18,905 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-14 07:48:18,905 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32134.41 MB 2025-02-14 07:48:19,081 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 07:48:19,081 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 07:48:19,081 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 07:48:19,081 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:48:19,081 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28123.67 MB 2025-02-14 07:48:19,081 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28890.67 MB 2025-02-14 07:48:19,081 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 07:48:19,081 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34256.98 MB 2025-02-14 07:48:19,081 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34670.12 MB 2025-02-14 07:48:19,081 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 07:48:19,081 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29598.46 MB 2025-02-14 07:48:19,100 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 07:48:19,101 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 07:48:19,101 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:48:19,101 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:48:19,101 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29303.56 MB 2025-02-14 07:48:19,101 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29530.76 MB 2025-02-14 07:48:19,101 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.19 MB 2025-02-14 07:48:19,101 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34670.12 MB 2025-02-14 07:48:19,101 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34670.12 MB 2025-02-14 07:48:19,101 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:48:19,101 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29774.29 MB 2025-02-14 07:48:19,102 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 07:48:19,102 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 07:48:19,102 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.83 seconds 2025-02-14 07:48:19,102 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:48:19,102 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17090.37 MB 2025-02-14 07:48:19,102 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29730.94 MB 2025-02-14 07:48:19,102 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12640.57 MB 2025-02-14 07:48:19,102 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56553.90 MB 2025-02-14 07:48:19,102 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34670.12 MB 2025-02-14 07:48:19,102 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21883.78 MB 2025-02-14 07:48:19,102 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29774.29 MB 2025-02-14 07:48:19,372 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 07:48:19,372 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 07:48:19,372 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 07:48:19,372 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:48:19,372 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29730.94 MB 2025-02-14 07:48:19,372 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22081.05 MB 2025-02-14 07:48:19,372 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7649.89 MB 2025-02-14 07:48:19,372 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34670.12 MB 2025-02-14 07:48:19,372 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34670.12 MB 2025-02-14 07:48:19,372 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:48:19,372 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32231.55 MB 2025-02-14 07:48:19,390 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8126, cut from 8128 2025-02-14 07:48:19,390 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 07:48:19,396 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 07:48:19,396 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 07:48:19,396 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:48:19,396 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:48:19,396 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22081.05 MB 2025-02-14 07:48:19,396 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30482.58 MB 2025-02-14 07:48:19,396 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8401.53 MB 2025-02-14 07:48:19,396 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34670.12 MB 2025-02-14 07:48:19,396 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43025.17 MB 2025-02-14 07:48:19,396 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8355.05 MB 2025-02-14 07:48:19,396 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30482.58 MB 2025-02-14 07:48:19,561 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7918] 2025-02-14 07:48:19,562 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:48:19,562 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 07:48:19,563 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:48:19,563 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 07:48:19,568 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 07:48:19,569 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:48:19,569 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 07:48:19,569 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 07:49:23,495 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:49:23,495 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 07:49:23,501 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 07:49:23,505 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:49:23,505 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1142, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 07:49:23,507 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:49:23,507 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1142, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 07:49:41,137 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 07:49:41,138 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 07:49:41,138 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.62 seconds 2025-02-14 07:49:41,138 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:49:41,138 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20926.35 MB 2025-02-14 07:49:41,138 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24967.82 MB 2025-02-14 07:49:41,138 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4041.47 MB 2025-02-14 07:49:41,138 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51380.22 MB 2025-02-14 07:49:41,138 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29334.96 MB 2025-02-14 07:49:41,138 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22045.26 MB 2025-02-14 07:49:41,138 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33795.91 MB 2025-02-14 07:49:41,214 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 07:49:41,214 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 07:49:41,214 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 07:49:41,214 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:49:41,214 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24967.82 MB 2025-02-14 07:49:41,214 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21714.75 MB 2025-02-14 07:49:41,214 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3253.07 MB 2025-02-14 07:49:41,214 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29334.96 MB 2025-02-14 07:49:41,215 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44161.83 MB 2025-02-14 07:49:41,215 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 14826.86 MB 2025-02-14 07:49:41,215 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37200.51 MB 2025-02-14 07:49:43,144 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 07:49:43,144 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 07:49:43,144 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 07:49:43,144 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:49:43,144 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21714.75 MB 2025-02-14 07:49:43,144 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22245.59 MB 2025-02-14 07:49:43,144 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 07:49:43,144 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44161.83 MB 2025-02-14 07:49:43,144 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26707.23 MB 2025-02-14 07:49:43,144 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17454.60 MB 2025-02-14 07:49:43,144 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26224.93 MB 2025-02-14 07:49:43,158 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 07:49:43,158 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 07:49:43,158 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:49:43,158 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:49:43,158 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22245.59 MB 2025-02-14 07:49:43,158 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24135.13 MB 2025-02-14 07:49:43,158 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 07:49:43,158 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26707.23 MB 2025-02-14 07:49:43,158 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28594.67 MB 2025-02-14 07:49:43,158 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 07:49:43,158 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25552.56 MB 2025-02-14 07:49:43,371 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 07:49:43,371 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 07:49:43,371 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 07:49:43,371 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:49:43,371 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24135.13 MB 2025-02-14 07:49:43,371 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26376.98 MB 2025-02-14 07:49:43,371 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 07:49:43,371 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28594.67 MB 2025-02-14 07:49:43,371 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34256.98 MB 2025-02-14 07:49:43,371 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 07:49:43,371 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31921.27 MB 2025-02-14 07:49:43,372 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 07:49:43,372 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 07:49:43,372 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 07:49:43,372 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:49:43,372 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22245.59 MB 2025-02-14 07:49:43,372 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26376.98 MB 2025-02-14 07:49:43,372 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 07:49:43,372 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26707.23 MB 2025-02-14 07:49:43,372 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34256.98 MB 2025-02-14 07:49:43,372 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-14 07:49:43,372 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31921.27 MB 2025-02-14 07:49:43,596 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 07:49:43,597 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 07:49:43,597 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 07:49:43,597 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:49:43,597 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27910.53 MB 2025-02-14 07:49:43,597 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28677.53 MB 2025-02-14 07:49:43,597 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 07:49:43,597 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34256.98 MB 2025-02-14 07:49:43,597 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34670.12 MB 2025-02-14 07:49:43,597 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 07:49:43,597 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29385.32 MB 2025-02-14 07:49:43,616 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 07:49:43,616 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 07:49:43,616 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:49:43,616 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:49:43,616 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29090.42 MB 2025-02-14 07:49:43,616 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29317.36 MB 2025-02-14 07:49:43,616 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.95 MB 2025-02-14 07:49:43,616 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34670.12 MB 2025-02-14 07:49:43,616 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34670.12 MB 2025-02-14 07:49:43,616 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:49:43,616 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29552.99 MB 2025-02-14 07:49:43,617 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 07:49:43,617 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 07:49:43,617 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.11 seconds 2025-02-14 07:49:43,617 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:49:43,617 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16947.53 MB 2025-02-14 07:49:43,617 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29517.30 MB 2025-02-14 07:49:43,617 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12569.78 MB 2025-02-14 07:49:43,617 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51380.22 MB 2025-02-14 07:49:43,617 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34670.12 MB 2025-02-14 07:49:43,617 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16710.11 MB 2025-02-14 07:49:43,617 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29552.99 MB 2025-02-14 07:49:43,886 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 07:49:43,886 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 07:49:43,886 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 07:49:43,886 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:49:43,886 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29517.30 MB 2025-02-14 07:49:43,886 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21934.39 MB 2025-02-14 07:49:43,886 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7582.91 MB 2025-02-14 07:49:43,886 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34670.12 MB 2025-02-14 07:49:43,886 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34670.12 MB 2025-02-14 07:49:43,886 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:49:43,886 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32014.84 MB 2025-02-14 07:49:43,904 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8116, cut from 8118 2025-02-14 07:49:43,904 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 07:49:43,910 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 07:49:43,910 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 07:49:43,910 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:49:43,910 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:49:43,910 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21934.39 MB 2025-02-14 07:49:43,910 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30326.82 MB 2025-02-14 07:49:43,910 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8392.42 MB 2025-02-14 07:49:43,910 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34670.12 MB 2025-02-14 07:49:43,910 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43012.59 MB 2025-02-14 07:49:43,910 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8342.47 MB 2025-02-14 07:49:43,910 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30326.82 MB 2025-02-14 07:49:44,072 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7908] 2025-02-14 07:49:44,073 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:49:44,073 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 07:49:44,074 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:49:44,074 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 07:49:44,079 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 07:49:44,080 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:49:44,080 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 07:49:44,080 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 07:51:03,877 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:51:03,877 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 07:51:03,882 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 07:51:03,886 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:51:03,886 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1581, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 07:51:03,887 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:51:03,887 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1581, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 07:51:28,263 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 07:51:28,264 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 07:51:28,264 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.37 seconds 2025-02-14 07:51:28,264 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:51:28,264 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23985.37 MB 2025-02-14 07:51:28,264 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29580.57 MB 2025-02-14 07:51:28,264 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5595.20 MB 2025-02-14 07:51:28,264 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51355.06 MB 2025-02-14 07:51:28,264 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39189.48 MB 2025-02-14 07:51:28,264 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12165.58 MB 2025-02-14 07:51:28,264 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38440.38 MB 2025-02-14 07:51:28,349 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 07:51:28,349 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 07:51:28,349 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 07:51:28,349 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:51:28,349 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29580.57 MB 2025-02-14 07:51:28,349 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23996.98 MB 2025-02-14 07:51:28,349 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5583.59 MB 2025-02-14 07:51:28,349 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39189.48 MB 2025-02-14 07:51:28,349 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48932.85 MB 2025-02-14 07:51:28,349 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9743.37 MB 2025-02-14 07:51:28,349 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44023.42 MB 2025-02-14 07:51:30,273 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 07:51:30,273 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 07:51:30,273 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 07:51:30,273 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:51:30,273 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23996.98 MB 2025-02-14 07:51:30,273 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24527.82 MB 2025-02-14 07:51:30,273 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 07:51:30,273 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48932.85 MB 2025-02-14 07:51:30,273 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29423.04 MB 2025-02-14 07:51:30,273 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19509.81 MB 2025-02-14 07:51:30,273 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28507.15 MB 2025-02-14 07:51:30,289 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 07:51:30,289 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 07:51:30,289 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:51:30,289 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:51:30,289 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24527.82 MB 2025-02-14 07:51:30,289 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26417.35 MB 2025-02-14 07:51:30,289 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 07:51:30,289 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29423.04 MB 2025-02-14 07:51:30,289 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30366.76 MB 2025-02-14 07:51:30,289 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 07:51:30,289 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27834.78 MB 2025-02-14 07:51:30,503 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 07:51:30,503 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 07:51:30,503 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 07:51:30,503 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:51:30,503 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26417.35 MB 2025-02-14 07:51:30,503 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28659.21 MB 2025-02-14 07:51:30,503 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 07:51:30,503 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30366.76 MB 2025-02-14 07:51:30,503 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36500.93 MB 2025-02-14 07:51:30,503 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 07:51:30,503 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34203.49 MB 2025-02-14 07:51:30,504 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 07:51:30,504 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 07:51:30,504 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 07:51:30,504 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:51:30,504 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24527.82 MB 2025-02-14 07:51:30,504 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28659.21 MB 2025-02-14 07:51:30,504 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 07:51:30,504 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29423.04 MB 2025-02-14 07:51:30,504 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36500.93 MB 2025-02-14 07:51:30,504 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7077.89 MB 2025-02-14 07:51:30,504 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34203.49 MB 2025-02-14 07:51:30,683 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 07:51:30,683 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 07:51:30,683 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 07:51:30,683 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:51:30,683 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30192.75 MB 2025-02-14 07:51:30,683 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30959.75 MB 2025-02-14 07:51:30,683 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 07:51:30,683 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36500.93 MB 2025-02-14 07:51:30,683 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36916.17 MB 2025-02-14 07:51:30,683 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 07:51:30,683 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31667.54 MB 2025-02-14 07:51:30,703 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 07:51:30,703 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 07:51:30,703 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:51:30,703 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:51:30,703 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31372.64 MB 2025-02-14 07:51:30,703 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31601.55 MB 2025-02-14 07:51:30,703 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.91 MB 2025-02-14 07:51:30,703 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36916.17 MB 2025-02-14 07:51:30,703 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36916.17 MB 2025-02-14 07:51:30,703 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:51:30,703 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31837.52 MB 2025-02-14 07:51:30,704 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 07:51:30,704 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 07:51:30,704 - resource_logging.py:150 - __exit__ - DEBUG - Time: 26.82 seconds 2025-02-14 07:51:30,704 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:51:30,704 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18477.04 MB 2025-02-14 07:51:30,704 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31802.38 MB 2025-02-14 07:51:30,704 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13325.34 MB 2025-02-14 07:51:30,704 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51355.06 MB 2025-02-14 07:51:30,704 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36916.17 MB 2025-02-14 07:51:30,704 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14438.89 MB 2025-02-14 07:51:30,704 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31837.52 MB 2025-02-14 07:51:30,974 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 07:51:30,974 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 07:51:30,974 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 07:51:30,974 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:51:30,974 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31802.38 MB 2025-02-14 07:51:30,974 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23477.62 MB 2025-02-14 07:51:30,974 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8324.76 MB 2025-02-14 07:51:30,974 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36916.17 MB 2025-02-14 07:51:30,974 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36916.17 MB 2025-02-14 07:51:30,974 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:51:30,974 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34310.97 MB 2025-02-14 07:51:30,992 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8152, cut from 8154 2025-02-14 07:51:30,992 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 07:51:30,998 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 07:51:30,998 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 07:51:30,998 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:51:30,998 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:51:30,999 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23477.62 MB 2025-02-14 07:51:30,999 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31906.74 MB 2025-02-14 07:51:30,999 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8429.12 MB 2025-02-14 07:51:30,999 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36916.17 MB 2025-02-14 07:51:30,999 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45296.39 MB 2025-02-14 07:51:30,999 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8380.22 MB 2025-02-14 07:51:30,999 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31906.74 MB 2025-02-14 07:51:31,168 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7944] 2025-02-14 07:51:31,169 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:51:31,169 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 07:51:31,170 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:51:31,170 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 07:51:31,175 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 07:51:31,176 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:51:31,176 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 07:51:31,176 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 07:53:59,059 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:53:59,060 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 07:53:59,065 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 07:53:59,070 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:53:59,070 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1986, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 07:53:59,071 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:53:59,071 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1986, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 07:54:29,650 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 07:54:29,651 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 07:54:29,651 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.57 seconds 2025-02-14 07:54:29,651 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:54:29,651 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26807.47 MB 2025-02-14 07:54:29,651 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33835.81 MB 2025-02-14 07:54:29,651 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7028.34 MB 2025-02-14 07:54:29,651 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53676.61 MB 2025-02-14 07:54:29,651 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40644.90 MB 2025-02-14 07:54:29,651 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13031.70 MB 2025-02-14 07:54:29,651 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42847.93 MB 2025-02-14 07:54:29,774 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 07:54:29,774 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 07:54:29,774 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 07:54:29,774 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:54:29,774 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33835.81 MB 2025-02-14 07:54:29,774 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26102.45 MB 2025-02-14 07:54:29,774 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7733.37 MB 2025-02-14 07:54:29,774 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40644.90 MB 2025-02-14 07:54:29,774 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 63245.91 MB 2025-02-14 07:54:29,774 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 22601.01 MB 2025-02-14 07:54:29,774 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 53565.30 MB 2025-02-14 07:54:31,711 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 07:54:31,711 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 07:54:31,711 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-14 07:54:31,711 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:54:31,711 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26102.45 MB 2025-02-14 07:54:31,711 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26633.29 MB 2025-02-14 07:54:31,711 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 07:54:31,711 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63245.91 MB 2025-02-14 07:54:31,711 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30859.59 MB 2025-02-14 07:54:31,711 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32386.32 MB 2025-02-14 07:54:31,711 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30613.66 MB 2025-02-14 07:54:31,725 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 07:54:31,725 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 07:54:31,725 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:54:31,725 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:54:31,725 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26633.29 MB 2025-02-14 07:54:31,725 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28522.82 MB 2025-02-14 07:54:31,725 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 07:54:31,725 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30859.59 MB 2025-02-14 07:54:31,725 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32747.03 MB 2025-02-14 07:54:31,725 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 07:54:31,725 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29940.25 MB 2025-02-14 07:54:31,934 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 07:54:31,934 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 07:54:31,934 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 07:54:31,934 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:54:31,934 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28522.82 MB 2025-02-14 07:54:31,934 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30764.68 MB 2025-02-14 07:54:31,934 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 07:54:31,934 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32747.03 MB 2025-02-14 07:54:31,934 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38881.20 MB 2025-02-14 07:54:31,934 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 07:54:31,934 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36308.96 MB 2025-02-14 07:54:31,935 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 07:54:31,935 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 07:54:31,935 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 07:54:31,935 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:54:31,935 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26633.29 MB 2025-02-14 07:54:31,935 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30764.68 MB 2025-02-14 07:54:31,935 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 07:54:31,935 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30859.59 MB 2025-02-14 07:54:31,935 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38881.20 MB 2025-02-14 07:54:31,935 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-14 07:54:31,935 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36308.96 MB 2025-02-14 07:54:32,105 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 07:54:32,105 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 07:54:32,105 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 07:54:32,105 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:54:32,105 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32298.22 MB 2025-02-14 07:54:32,105 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33065.22 MB 2025-02-14 07:54:32,105 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 07:54:32,105 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38881.20 MB 2025-02-14 07:54:32,105 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39294.34 MB 2025-02-14 07:54:32,105 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 07:54:32,105 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33773.01 MB 2025-02-14 07:54:32,125 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 07:54:32,125 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 07:54:32,125 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:54:32,125 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:54:32,125 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33478.11 MB 2025-02-14 07:54:32,125 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33706.33 MB 2025-02-14 07:54:32,125 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.22 MB 2025-02-14 07:54:32,125 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39294.34 MB 2025-02-14 07:54:32,125 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39294.34 MB 2025-02-14 07:54:32,125 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:54:32,125 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33911.64 MB 2025-02-14 07:54:32,126 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 07:54:32,126 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 07:54:32,126 - resource_logging.py:150 - __exit__ - DEBUG - Time: 33.05 seconds 2025-02-14 07:54:32,126 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:54:32,126 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19888.09 MB 2025-02-14 07:54:32,126 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33906.47 MB 2025-02-14 07:54:32,126 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14018.38 MB 2025-02-14 07:54:32,126 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53676.61 MB 2025-02-14 07:54:32,126 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39294.34 MB 2025-02-14 07:54:32,126 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14382.27 MB 2025-02-14 07:54:32,126 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33911.64 MB 2025-02-14 07:54:32,394 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 07:54:32,394 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 07:54:32,395 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 07:54:32,395 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:54:32,395 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33906.47 MB 2025-02-14 07:54:32,395 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24878.00 MB 2025-02-14 07:54:32,395 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9028.47 MB 2025-02-14 07:54:32,395 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39294.34 MB 2025-02-14 07:54:32,395 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39294.34 MB 2025-02-14 07:54:32,395 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:54:32,395 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36406.47 MB 2025-02-14 07:54:32,412 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8124, cut from 8126 2025-02-14 07:54:32,413 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 07:54:32,419 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 07:54:32,419 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 07:54:32,419 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:54:32,419 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:54:32,419 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24878.00 MB 2025-02-14 07:54:32,419 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33278.86 MB 2025-02-14 07:54:32,419 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8400.86 MB 2025-02-14 07:54:32,419 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39294.34 MB 2025-02-14 07:54:32,419 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47645.20 MB 2025-02-14 07:54:32,419 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8350.86 MB 2025-02-14 07:54:32,419 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33278.86 MB 2025-02-14 07:54:32,588 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7916] 2025-02-14 07:54:32,589 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:54:32,589 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 07:54:32,590 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:54:32,590 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 07:54:32,595 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 07:54:32,596 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:54:32,596 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 07:54:32,596 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 07:56:58,763 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:56:58,763 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 07:56:58,769 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 07:56:58,773 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:56:58,774 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 3473, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 07:56:58,774 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:56:58,775 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 3473, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 07:57:52,412 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 07:57:52,412 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 07:57:52,412 - resource_logging.py:150 - __exit__ - DEBUG - Time: 53.62 seconds 2025-02-14 07:57:52,412 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:57:52,412 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37169.13 MB 2025-02-14 07:57:52,412 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 49460.53 MB 2025-02-14 07:57:52,412 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12291.41 MB 2025-02-14 07:57:52,412 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 80201.38 MB 2025-02-14 07:57:52,413 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53406.07 MB 2025-02-14 07:57:52,413 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26795.31 MB 2025-02-14 07:57:52,413 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 61751.94 MB 2025-02-14 07:57:52,646 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 07:57:52,646 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 07:57:52,646 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 07:57:52,646 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:57:52,646 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 49460.53 MB 2025-02-14 07:57:52,646 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33832.89 MB 2025-02-14 07:57:52,646 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -15627.64 MB 2025-02-14 07:57:52,646 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53406.07 MB 2025-02-14 07:57:52,646 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 98324.97 MB 2025-02-14 07:57:52,646 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 44918.90 MB 2025-02-14 07:57:52,646 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 83555.46 MB 2025-02-14 07:57:54,644 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 07:57:54,645 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 07:57:54,645 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.00 seconds 2025-02-14 07:57:54,645 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:57:54,645 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33832.89 MB 2025-02-14 07:57:54,645 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34363.74 MB 2025-02-14 07:57:54,645 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 07:57:54,645 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 98324.97 MB 2025-02-14 07:57:54,645 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36383.49 MB 2025-02-14 07:57:54,645 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -61941.48 MB 2025-02-14 07:57:54,645 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38344.11 MB 2025-02-14 07:57:54,659 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 07:57:54,659 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 07:57:54,659 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:57:54,659 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:57:54,659 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34363.74 MB 2025-02-14 07:57:54,659 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36253.27 MB 2025-02-14 07:57:54,659 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 07:57:54,659 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36383.49 MB 2025-02-14 07:57:54,659 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39686.50 MB 2025-02-14 07:57:54,659 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 07:57:54,659 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37670.70 MB 2025-02-14 07:57:54,869 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 07:57:54,869 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 07:57:54,869 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 07:57:54,869 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:57:54,869 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36253.27 MB 2025-02-14 07:57:54,869 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38495.13 MB 2025-02-14 07:57:54,869 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 07:57:54,869 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39686.50 MB 2025-02-14 07:57:54,869 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46292.53 MB 2025-02-14 07:57:54,869 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 07:57:54,869 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44039.41 MB 2025-02-14 07:57:54,870 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 07:57:54,870 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 07:57:54,870 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 07:57:54,870 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:57:54,870 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34363.74 MB 2025-02-14 07:57:54,870 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38495.13 MB 2025-02-14 07:57:54,870 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 07:57:54,870 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36383.49 MB 2025-02-14 07:57:54,870 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46292.53 MB 2025-02-14 07:57:54,870 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9909.04 MB 2025-02-14 07:57:54,870 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44039.41 MB 2025-02-14 07:57:55,040 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 07:57:55,040 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 07:57:55,040 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 07:57:55,040 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:57:55,040 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40028.67 MB 2025-02-14 07:57:55,040 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40795.67 MB 2025-02-14 07:57:55,040 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 07:57:55,040 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46292.53 MB 2025-02-14 07:57:55,040 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46707.77 MB 2025-02-14 07:57:55,040 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 07:57:55,040 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41503.46 MB 2025-02-14 07:57:55,059 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 07:57:55,059 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 07:57:55,059 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:57:55,059 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:57:55,059 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41208.56 MB 2025-02-14 07:57:55,059 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41437.46 MB 2025-02-14 07:57:55,059 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.90 MB 2025-02-14 07:57:55,059 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46707.77 MB 2025-02-14 07:57:55,059 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46707.77 MB 2025-02-14 07:57:55,060 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:57:55,060 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41652.51 MB 2025-02-14 07:57:55,061 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 07:57:55,061 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 07:57:55,061 - resource_logging.py:150 - __exit__ - DEBUG - Time: 56.28 seconds 2025-02-14 07:57:55,061 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:57:55,061 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25068.92 MB 2025-02-14 07:57:55,061 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41638.11 MB 2025-02-14 07:57:55,061 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 16569.20 MB 2025-02-14 07:57:55,061 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 68098.72 MB 2025-02-14 07:57:55,061 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46707.77 MB 2025-02-14 07:57:55,061 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21390.95 MB 2025-02-14 07:57:55,061 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41652.51 MB 2025-02-14 07:57:55,331 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 07:57:55,331 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 07:57:55,331 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 07:57:55,331 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:57:55,331 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41638.11 MB 2025-02-14 07:57:55,331 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30066.83 MB 2025-02-14 07:57:55,331 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -11571.28 MB 2025-02-14 07:57:55,331 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46707.77 MB 2025-02-14 07:57:55,331 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46707.77 MB 2025-02-14 07:57:55,331 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:57:55,331 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44144.56 MB 2025-02-14 07:57:55,349 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8145, cut from 8147 2025-02-14 07:57:55,350 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-14 07:57:55,355 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 07:57:55,355 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 07:57:55,355 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:57:55,355 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:57:55,355 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30066.83 MB 2025-02-14 07:57:55,355 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38488.13 MB 2025-02-14 07:57:55,355 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8421.30 MB 2025-02-14 07:57:55,355 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46707.77 MB 2025-02-14 07:57:55,355 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50893.68 MB 2025-02-14 07:57:55,355 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4185.92 MB 2025-02-14 07:57:55,355 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38488.13 MB 2025-02-14 07:57:55,518 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7937] 2025-02-14 07:57:55,519 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:57:55,519 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 07:57:55,520 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:57:55,520 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 07:57:55,525 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 07:57:55,526 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:57:55,526 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 07:57:55,526 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-14 07:58:08,003 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:58:08,003 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 07:58:08,009 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 07:58:08,012 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:58:08,012 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 3305, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 07:58:08,013 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:58:08,013 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 3305, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 07:58:59,961 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 07:58:59,962 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 07:58:59,962 - resource_logging.py:150 - __exit__ - DEBUG - Time: 51.94 seconds 2025-02-14 07:58:59,962 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:58:59,962 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35998.48 MB 2025-02-14 07:58:59,962 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47694.69 MB 2025-02-14 07:58:59,962 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11696.21 MB 2025-02-14 07:58:59,962 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64934.12 MB 2025-02-14 07:58:59,962 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52332.33 MB 2025-02-14 07:58:59,962 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12601.79 MB 2025-02-14 07:58:59,962 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 59390.90 MB 2025-02-14 07:59:00,227 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 07:59:00,227 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 07:59:00,227 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 07:59:00,227 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:59:00,227 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47694.69 MB 2025-02-14 07:59:00,227 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32959.51 MB 2025-02-14 07:59:00,227 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -14735.17 MB 2025-02-14 07:59:00,227 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52332.33 MB 2025-02-14 07:59:00,227 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 96806.63 MB 2025-02-14 07:59:00,227 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 44474.30 MB 2025-02-14 07:59:00,227 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 82002.33 MB 2025-02-14 07:59:02,237 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 07:59:02,237 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 07:59:02,237 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.01 seconds 2025-02-14 07:59:02,237 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:59:02,237 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32959.51 MB 2025-02-14 07:59:02,237 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33490.36 MB 2025-02-14 07:59:02,237 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 07:59:02,237 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 96806.63 MB 2025-02-14 07:59:02,237 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36094.08 MB 2025-02-14 07:59:02,237 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -60712.55 MB 2025-02-14 07:59:02,238 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37470.73 MB 2025-02-14 07:59:02,252 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 07:59:02,252 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 07:59:02,252 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 07:59:02,252 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:59:02,252 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33490.36 MB 2025-02-14 07:59:02,252 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35379.62 MB 2025-02-14 07:59:02,252 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.26 MB 2025-02-14 07:59:02,252 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36094.08 MB 2025-02-14 07:59:02,252 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38925.24 MB 2025-02-14 07:59:02,252 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-14 07:59:02,252 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36797.04 MB 2025-02-14 07:59:02,465 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 07:59:02,465 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 07:59:02,465 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 07:59:02,465 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:59:02,465 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35379.62 MB 2025-02-14 07:59:02,465 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37621.47 MB 2025-02-14 07:59:02,465 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 07:59:02,465 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38925.24 MB 2025-02-14 07:59:02,465 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45531.27 MB 2025-02-14 07:59:02,465 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 07:59:02,465 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43165.75 MB 2025-02-14 07:59:02,465 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 07:59:02,465 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 07:59:02,465 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 07:59:02,466 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:59:02,466 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33490.36 MB 2025-02-14 07:59:02,466 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37621.47 MB 2025-02-14 07:59:02,466 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.12 MB 2025-02-14 07:59:02,466 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36094.08 MB 2025-02-14 07:59:02,466 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45531.27 MB 2025-02-14 07:59:02,466 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9437.18 MB 2025-02-14 07:59:02,466 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43165.75 MB 2025-02-14 07:59:02,633 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 07:59:02,633 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 07:59:02,633 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 07:59:02,633 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:59:02,633 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39155.01 MB 2025-02-14 07:59:02,633 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39922.02 MB 2025-02-14 07:59:02,633 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 07:59:02,633 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45531.27 MB 2025-02-14 07:59:02,633 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45948.60 MB 2025-02-14 07:59:02,633 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 07:59:02,633 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40629.80 MB 2025-02-14 07:59:02,652 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 07:59:02,652 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 07:59:02,652 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:59:02,652 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:59:02,652 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40334.90 MB 2025-02-14 07:59:02,652 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40563.30 MB 2025-02-14 07:59:02,652 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.40 MB 2025-02-14 07:59:02,652 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45948.60 MB 2025-02-14 07:59:02,652 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45948.60 MB 2025-02-14 07:59:02,652 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:59:02,652 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40781.54 MB 2025-02-14 07:59:02,653 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 07:59:02,653 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 07:59:02,653 - resource_logging.py:150 - __exit__ - DEBUG - Time: 54.64 seconds 2025-02-14 07:59:02,653 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:59:02,653 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24483.59 MB 2025-02-14 07:59:02,653 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40763.61 MB 2025-02-14 07:59:02,653 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 16280.02 MB 2025-02-14 07:59:02,653 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64934.12 MB 2025-02-14 07:59:02,653 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45948.60 MB 2025-02-14 07:59:02,653 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18985.52 MB 2025-02-14 07:59:02,653 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40781.54 MB 2025-02-14 07:59:02,923 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 07:59:02,923 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 07:59:02,923 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 07:59:02,923 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:59:02,923 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40763.61 MB 2025-02-14 07:59:02,923 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29476.17 MB 2025-02-14 07:59:02,923 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -11287.44 MB 2025-02-14 07:59:02,923 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45948.60 MB 2025-02-14 07:59:02,923 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45948.60 MB 2025-02-14 07:59:02,923 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 07:59:02,923 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43265.75 MB 2025-02-14 07:59:02,941 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8131, cut from 8133 2025-02-14 07:59:02,942 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 07:59:02,947 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 07:59:02,947 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 07:59:02,947 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 07:59:02,947 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 07:59:02,947 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29476.17 MB 2025-02-14 07:59:02,947 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37883.91 MB 2025-02-14 07:59:02,947 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8407.74 MB 2025-02-14 07:59:02,947 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45948.60 MB 2025-02-14 07:59:02,947 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50128.22 MB 2025-02-14 07:59:02,947 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4179.62 MB 2025-02-14 07:59:02,947 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37883.91 MB 2025-02-14 07:59:03,109 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7923] 2025-02-14 07:59:03,111 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:59:03,111 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 07:59:03,112 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:59:03,112 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 07:59:03,116 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 07:59:03,117 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:59:03,118 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 07:59:03,118 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 07:59:59,836 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:59:59,836 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 07:59:59,841 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 07:59:59,845 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:59:59,845 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 283, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 07:59:59,846 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 07:59:59,846 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 283, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 08:00:04,264 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 08:00:04,264 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 08:00:04,264 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.41 seconds 2025-02-14 08:00:04,264 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:00:04,264 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14940.70 MB 2025-02-14 08:00:04,265 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15942.22 MB 2025-02-14 08:00:04,265 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1001.52 MB 2025-02-14 08:00:04,265 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58487.47 MB 2025-02-14 08:00:04,265 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18792.58 MB 2025-02-14 08:00:04,265 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -39694.89 MB 2025-02-14 08:00:04,265 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24865.86 MB 2025-02-14 08:00:04,276 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 08:00:04,276 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 08:00:04,276 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:00:04,276 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:00:04,276 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15942.22 MB 2025-02-14 08:00:04,276 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14580.40 MB 2025-02-14 08:00:04,276 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1361.82 MB 2025-02-14 08:00:04,276 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18792.58 MB 2025-02-14 08:00:04,276 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18792.58 MB 2025-02-14 08:00:04,276 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:00:04,276 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16262.13 MB 2025-02-14 08:00:04,381 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 08:00:04,381 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 08:00:04,381 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 08:00:04,381 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:00:04,381 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14580.40 MB 2025-02-14 08:00:04,381 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14606.94 MB 2025-02-14 08:00:04,381 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 26.54 MB 2025-02-14 08:00:04,381 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18792.58 MB 2025-02-14 08:00:04,381 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18792.58 MB 2025-02-14 08:00:04,381 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:00:04,381 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15856.87 MB 2025-02-14 08:00:04,385 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 08:00:04,385 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 08:00:04,385 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 08:00:04,385 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:00:04,385 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14606.87 MB 2025-02-14 08:00:04,385 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14701.33 MB 2025-02-14 08:00:04,386 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 94.45 MB 2025-02-14 08:00:04,386 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18792.58 MB 2025-02-14 08:00:04,386 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18792.58 MB 2025-02-14 08:00:04,386 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:00:04,386 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14772.21 MB 2025-02-14 08:00:04,402 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 08:00:04,402 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 08:00:04,402 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:00:04,402 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:00:04,402 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14701.33 MB 2025-02-14 08:00:04,402 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14813.87 MB 2025-02-14 08:00:04,402 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 112.55 MB 2025-02-14 08:00:04,402 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18792.58 MB 2025-02-14 08:00:04,402 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18792.58 MB 2025-02-14 08:00:04,402 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:00:04,402 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15091.70 MB 2025-02-14 08:00:04,403 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 08:00:04,403 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 08:00:04,403 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:00:04,403 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:00:04,403 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14606.87 MB 2025-02-14 08:00:04,403 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14813.87 MB 2025-02-14 08:00:04,403 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 207.00 MB 2025-02-14 08:00:04,403 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18792.58 MB 2025-02-14 08:00:04,403 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18792.58 MB 2025-02-14 08:00:04,403 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:00:04,403 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15091.70 MB 2025-02-14 08:00:04,414 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 08:00:04,414 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 08:00:04,414 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:00:04,414 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:00:04,415 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14891.23 MB 2025-02-14 08:00:04,415 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14929.58 MB 2025-02-14 08:00:04,415 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 38.35 MB 2025-02-14 08:00:04,415 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18792.58 MB 2025-02-14 08:00:04,415 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18813.55 MB 2025-02-14 08:00:04,415 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 20.97 MB 2025-02-14 08:00:04,415 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14980.37 MB 2025-02-14 08:00:04,417 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 08:00:04,417 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 08:00:04,417 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 08:00:04,417 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:00:04,417 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14950.23 MB 2025-02-14 08:00:04,417 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14975.85 MB 2025-02-14 08:00:04,417 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 25.62 MB 2025-02-14 08:00:04,417 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18813.55 MB 2025-02-14 08:00:04,417 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18813.55 MB 2025-02-14 08:00:04,417 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:00:04,417 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14975.85 MB 2025-02-14 08:00:04,418 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 08:00:04,418 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 08:00:04,418 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.57 seconds 2025-02-14 08:00:04,418 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:00:04,418 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13954.70 MB 2025-02-14 08:00:04,418 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15022.99 MB 2025-02-14 08:00:04,418 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1068.29 MB 2025-02-14 08:00:04,418 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58487.47 MB 2025-02-14 08:00:04,418 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18813.55 MB 2025-02-14 08:00:04,418 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -39673.92 MB 2025-02-14 08:00:04,418 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15022.99 MB 2025-02-14 08:00:04,490 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 08:00:04,490 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 08:00:04,490 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 08:00:04,490 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:00:04,490 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15022.99 MB 2025-02-14 08:00:04,490 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15729.68 MB 2025-02-14 08:00:04,490 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 706.69 MB 2025-02-14 08:00:04,490 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18813.55 MB 2025-02-14 08:00:04,490 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18817.74 MB 2025-02-14 08:00:04,491 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-14 08:00:04,491 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15800.34 MB 2025-02-14 08:00:04,496 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 1903, cut from 1905 2025-02-14 08:00:04,496 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 08:00:04,498 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 08:00:04,498 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 08:00:04,498 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:00:04,498 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:00:04,498 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14782.07 MB 2025-02-14 08:00:04,498 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16760.88 MB 2025-02-14 08:00:04,498 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1978.81 MB 2025-02-14 08:00:04,498 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18817.74 MB 2025-02-14 08:00:04,498 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19801.31 MB 2025-02-14 08:00:04,498 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 983.56 MB 2025-02-14 08:00:04,498 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16760.88 MB 2025-02-14 08:00:04,537 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 1695] 2025-02-14 08:00:04,538 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:00:04,538 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 08:00:04,539 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:00:04,539 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 08:00:04,544 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 08:00:04,545 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:00:04,545 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 08:00:04,545 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 08:01:10,288 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:01:10,288 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 08:01:10,293 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 08:01:10,297 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:01:10,297 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1265, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 08:01:10,298 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:01:10,298 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1265, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 08:01:29,792 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 08:01:29,792 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 08:01:29,792 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.49 seconds 2025-02-14 08:01:29,792 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:01:29,792 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21783.43 MB 2025-02-14 08:01:29,792 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26260.85 MB 2025-02-14 08:01:29,792 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4477.42 MB 2025-02-14 08:01:29,792 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30589.06 MB 2025-02-14 08:01:29,792 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30203.18 MB 2025-02-14 08:01:29,792 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -385.88 MB 2025-02-14 08:01:29,792 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35105.98 MB 2025-02-14 08:01:29,879 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 08:01:29,879 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 08:01:29,879 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 08:01:29,879 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:01:29,879 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26260.85 MB 2025-02-14 08:01:29,879 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22354.19 MB 2025-02-14 08:01:29,879 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3906.66 MB 2025-02-14 08:01:29,879 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30203.18 MB 2025-02-14 08:01:29,879 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47227.86 MB 2025-02-14 08:01:29,879 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 17024.68 MB 2025-02-14 08:01:29,879 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39625.97 MB 2025-02-14 08:01:31,817 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 08:01:31,817 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 08:01:31,817 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-14 08:01:31,817 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:01:31,817 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22354.19 MB 2025-02-14 08:01:31,817 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22885.03 MB 2025-02-14 08:01:31,817 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 08:01:31,817 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47227.86 MB 2025-02-14 08:01:31,817 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24901.58 MB 2025-02-14 08:01:31,817 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22326.28 MB 2025-02-14 08:01:31,817 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26865.40 MB 2025-02-14 08:01:31,831 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 08:01:31,831 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 08:01:31,831 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:01:31,831 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:01:31,831 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22885.03 MB 2025-02-14 08:01:31,831 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24774.57 MB 2025-02-14 08:01:31,831 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 08:01:31,831 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24901.58 MB 2025-02-14 08:01:31,831 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28204.60 MB 2025-02-14 08:01:31,831 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 08:01:31,831 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26192.00 MB 2025-02-14 08:01:32,040 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 08:01:32,040 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 08:01:32,040 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 08:01:32,040 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:01:32,040 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24774.57 MB 2025-02-14 08:01:32,040 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27016.42 MB 2025-02-14 08:01:32,040 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 08:01:32,040 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28204.60 MB 2025-02-14 08:01:32,040 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34812.72 MB 2025-02-14 08:01:32,040 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6608.13 MB 2025-02-14 08:01:32,040 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32560.70 MB 2025-02-14 08:01:32,041 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 08:01:32,041 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 08:01:32,041 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 08:01:32,041 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:01:32,041 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22885.03 MB 2025-02-14 08:01:32,041 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27016.42 MB 2025-02-14 08:01:32,041 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 08:01:32,041 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24901.58 MB 2025-02-14 08:01:32,041 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34812.72 MB 2025-02-14 08:01:32,041 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9911.14 MB 2025-02-14 08:01:32,041 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32560.70 MB 2025-02-14 08:01:32,209 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 08:01:32,209 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 08:01:32,209 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 08:01:32,209 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:01:32,209 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28549.97 MB 2025-02-14 08:01:32,209 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29316.97 MB 2025-02-14 08:01:32,209 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 08:01:32,209 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34812.72 MB 2025-02-14 08:01:32,209 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35230.06 MB 2025-02-14 08:01:32,209 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 08:01:32,209 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30024.76 MB 2025-02-14 08:01:32,228 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 08:01:32,228 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 08:01:32,228 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:01:32,228 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:01:32,228 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29729.86 MB 2025-02-14 08:01:32,228 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29957.28 MB 2025-02-14 08:01:32,228 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.43 MB 2025-02-14 08:01:32,228 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35230.06 MB 2025-02-14 08:01:32,228 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35230.06 MB 2025-02-14 08:01:32,228 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:01:32,228 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30180.54 MB 2025-02-14 08:01:32,229 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 08:01:32,230 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 08:01:32,230 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.93 seconds 2025-02-14 08:01:32,230 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:01:32,230 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17376.07 MB 2025-02-14 08:01:32,230 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30157.42 MB 2025-02-14 08:01:32,230 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12781.36 MB 2025-02-14 08:01:32,230 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26178.75 MB 2025-02-14 08:01:32,230 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35230.06 MB 2025-02-14 08:01:32,230 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9051.31 MB 2025-02-14 08:01:32,230 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30180.54 MB 2025-02-14 08:01:32,499 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 08:01:32,499 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 08:01:32,499 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 08:01:32,499 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:01:32,499 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30157.42 MB 2025-02-14 08:01:32,499 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22365.46 MB 2025-02-14 08:01:32,499 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7791.97 MB 2025-02-14 08:01:32,499 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35230.06 MB 2025-02-14 08:01:32,499 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35230.06 MB 2025-02-14 08:01:32,499 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:01:32,499 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32657.42 MB 2025-02-14 08:01:32,517 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8124, cut from 8126 2025-02-14 08:01:32,517 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 08:01:32,523 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 08:01:32,523 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 08:01:32,523 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:01:32,523 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:01:32,523 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22365.46 MB 2025-02-14 08:01:32,523 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30766.32 MB 2025-02-14 08:01:32,523 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8400.86 MB 2025-02-14 08:01:32,524 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35230.06 MB 2025-02-14 08:01:32,524 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43580.92 MB 2025-02-14 08:01:32,524 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8350.86 MB 2025-02-14 08:01:32,524 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30766.32 MB 2025-02-14 08:01:32,687 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7916] 2025-02-14 08:01:32,688 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:01:32,688 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 08:01:32,689 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:01:32,689 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 08:01:32,694 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 08:01:32,695 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:01:32,695 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 08:01:32,695 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 08:01:43,317 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:01:43,317 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 08:01:43,322 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 08:01:43,325 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:01:43,325 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1683, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 08:01:43,326 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:01:43,326 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1683, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 08:02:09,588 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 08:02:09,588 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 08:02:09,588 - resource_logging.py:150 - __exit__ - DEBUG - Time: 26.25 seconds 2025-02-14 08:02:09,588 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:02:09,588 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24696.12 MB 2025-02-14 08:02:09,588 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30652.16 MB 2025-02-14 08:02:09,588 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5956.04 MB 2025-02-14 08:02:09,588 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51931.77 MB 2025-02-14 08:02:09,588 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39566.97 MB 2025-02-14 08:02:09,588 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12364.81 MB 2025-02-14 08:02:09,588 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39604.12 MB 2025-02-14 08:02:09,697 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 08:02:09,697 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 08:02:09,698 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 08:02:09,698 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:02:09,698 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30652.16 MB 2025-02-14 08:02:09,698 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24527.24 MB 2025-02-14 08:02:09,698 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6124.92 MB 2025-02-14 08:02:09,698 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39566.97 MB 2025-02-14 08:02:09,698 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56358.86 MB 2025-02-14 08:02:09,698 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 16791.90 MB 2025-02-14 08:02:09,698 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47651.34 MB 2025-02-14 08:02:11,641 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 08:02:11,641 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 08:02:11,641 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-14 08:02:11,641 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:02:11,641 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24527.24 MB 2025-02-14 08:02:11,641 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25058.08 MB 2025-02-14 08:02:11,641 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 08:02:11,641 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56358.86 MB 2025-02-14 08:02:11,641 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30849.11 MB 2025-02-14 08:02:11,641 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25509.76 MB 2025-02-14 08:02:11,641 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29037.42 MB 2025-02-14 08:02:11,654 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 08:02:11,654 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 08:02:11,654 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:02:11,654 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:02:11,654 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25058.08 MB 2025-02-14 08:02:11,654 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26947.62 MB 2025-02-14 08:02:11,654 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 08:02:11,654 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30849.11 MB 2025-02-14 08:02:11,654 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30849.11 MB 2025-02-14 08:02:11,654 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:02:11,654 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28365.05 MB 2025-02-14 08:02:11,867 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 08:02:11,868 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 08:02:11,868 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 08:02:11,868 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:02:11,868 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26947.62 MB 2025-02-14 08:02:11,868 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29189.47 MB 2025-02-14 08:02:11,868 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 08:02:11,868 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30849.11 MB 2025-02-14 08:02:11,868 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37455.13 MB 2025-02-14 08:02:11,868 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 08:02:11,868 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34733.76 MB 2025-02-14 08:02:11,868 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 08:02:11,868 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 08:02:11,868 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 08:02:11,868 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:02:11,868 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25058.08 MB 2025-02-14 08:02:11,868 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29189.47 MB 2025-02-14 08:02:11,868 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 08:02:11,868 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30849.11 MB 2025-02-14 08:02:11,868 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37455.13 MB 2025-02-14 08:02:11,868 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 08:02:11,868 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34733.76 MB 2025-02-14 08:02:12,037 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 08:02:12,037 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 08:02:12,037 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 08:02:12,038 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:02:12,038 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30723.02 MB 2025-02-14 08:02:12,038 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31490.02 MB 2025-02-14 08:02:12,038 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 08:02:12,038 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37455.13 MB 2025-02-14 08:02:12,038 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37872.47 MB 2025-02-14 08:02:12,038 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 08:02:12,038 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32197.81 MB 2025-02-14 08:02:12,056 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 08:02:12,056 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 08:02:12,056 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:02:12,056 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:02:12,056 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31902.91 MB 2025-02-14 08:02:12,057 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32131.77 MB 2025-02-14 08:02:12,057 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.86 MB 2025-02-14 08:02:12,057 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37872.47 MB 2025-02-14 08:02:12,057 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37872.47 MB 2025-02-14 08:02:12,057 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:02:12,057 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32347.35 MB 2025-02-14 08:02:12,058 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 08:02:12,058 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 08:02:12,058 - resource_logging.py:150 - __exit__ - DEBUG - Time: 28.73 seconds 2025-02-14 08:02:12,058 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:02:12,058 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18832.41 MB 2025-02-14 08:02:12,058 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32332.55 MB 2025-02-14 08:02:12,058 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13500.13 MB 2025-02-14 08:02:12,058 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51931.77 MB 2025-02-14 08:02:12,058 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37872.47 MB 2025-02-14 08:02:12,058 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14059.31 MB 2025-02-14 08:02:12,058 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32347.35 MB 2025-02-14 08:02:12,329 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 08:02:12,329 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 08:02:12,329 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 08:02:12,329 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:02:12,329 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32332.55 MB 2025-02-14 08:02:12,329 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23832.23 MB 2025-02-14 08:02:12,329 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8500.32 MB 2025-02-14 08:02:12,329 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37872.47 MB 2025-02-14 08:02:12,329 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37872.47 MB 2025-02-14 08:02:12,329 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:02:12,329 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34840.53 MB 2025-02-14 08:02:12,347 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8150, cut from 8152 2025-02-14 08:02:12,348 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 08:02:12,354 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 08:02:12,354 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 08:02:12,354 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:02:12,354 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:02:12,354 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23832.23 MB 2025-02-14 08:02:12,354 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32258.73 MB 2025-02-14 08:02:12,354 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8426.50 MB 2025-02-14 08:02:12,354 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37872.47 MB 2025-02-14 08:02:12,354 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46250.59 MB 2025-02-14 08:02:12,354 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8378.12 MB 2025-02-14 08:02:12,354 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32258.73 MB 2025-02-14 08:02:12,516 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7942] 2025-02-14 08:02:12,517 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:02:12,517 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 08:02:12,518 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:02:12,518 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 08:02:12,523 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 08:02:12,524 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:02:12,524 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 08:02:12,524 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 08:02:56,004 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:02:56,004 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 08:02:56,012 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 08:02:56,018 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:02:56,018 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 204, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 08:02:56,020 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:02:56,020 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 204, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 08:02:59,310 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 08:02:59,310 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 08:02:59,310 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.28 seconds 2025-02-14 08:02:59,310 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:02:59,310 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14390.21 MB 2025-02-14 08:02:59,310 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15112.16 MB 2025-02-14 08:02:59,310 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 721.94 MB 2025-02-14 08:02:59,310 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58816.72 MB 2025-02-14 08:02:59,310 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17853.05 MB 2025-02-14 08:02:59,310 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -40963.67 MB 2025-02-14 08:02:59,310 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24088.88 MB 2025-02-14 08:02:59,329 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 08:02:59,329 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 08:02:59,329 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:02:59,330 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:02:59,330 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15112.16 MB 2025-02-14 08:02:59,330 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15385.34 MB 2025-02-14 08:02:59,330 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 273.18 MB 2025-02-14 08:02:59,330 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17853.05 MB 2025-02-14 08:02:59,330 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19220.40 MB 2025-02-14 08:02:59,330 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1367.34 MB 2025-02-14 08:02:59,330 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17866.89 MB 2025-02-14 08:03:00,304 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 08:03:00,304 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 08:03:00,304 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.97 seconds 2025-02-14 08:03:00,304 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:03:00,304 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15385.34 MB 2025-02-14 08:03:00,304 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15641.47 MB 2025-02-14 08:03:00,304 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 256.13 MB 2025-02-14 08:03:00,304 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19220.40 MB 2025-02-14 08:03:00,304 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18467.52 MB 2025-02-14 08:03:00,304 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -752.88 MB 2025-02-14 08:03:00,304 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19641.75 MB 2025-02-14 08:03:00,316 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 08:03:00,316 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 08:03:00,316 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:03:00,316 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:03:00,316 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15641.40 MB 2025-02-14 08:03:00,316 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16552.88 MB 2025-02-14 08:03:00,316 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 911.48 MB 2025-02-14 08:03:00,316 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18467.52 MB 2025-02-14 08:03:00,316 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18924.70 MB 2025-02-14 08:03:00,316 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 457.18 MB 2025-02-14 08:03:00,316 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17236.80 MB 2025-02-14 08:03:00,457 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 08:03:00,457 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 08:03:00,457 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-14 08:03:00,457 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:03:00,458 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16552.88 MB 2025-02-14 08:03:00,458 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17635.10 MB 2025-02-14 08:03:00,458 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1082.22 MB 2025-02-14 08:03:00,458 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18924.70 MB 2025-02-14 08:03:00,458 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21667.77 MB 2025-02-14 08:03:00,458 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2743.07 MB 2025-02-14 08:03:00,458 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20310.19 MB 2025-02-14 08:03:00,459 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 08:03:00,459 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 08:03:00,459 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 08:03:00,459 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:03:00,459 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15641.40 MB 2025-02-14 08:03:00,459 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17635.10 MB 2025-02-14 08:03:00,459 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1993.70 MB 2025-02-14 08:03:00,459 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18467.52 MB 2025-02-14 08:03:00,459 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21667.77 MB 2025-02-14 08:03:00,459 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3200.25 MB 2025-02-14 08:03:00,459 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20310.19 MB 2025-02-14 08:03:00,596 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 08:03:00,596 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 08:03:00,596 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 08:03:00,596 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:03:00,596 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18375.04 MB 2025-02-14 08:03:00,596 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18745.12 MB 2025-02-14 08:03:00,596 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 370.08 MB 2025-02-14 08:03:00,596 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21667.77 MB 2025-02-14 08:03:00,596 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21867.00 MB 2025-02-14 08:03:00,596 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 199.23 MB 2025-02-14 08:03:00,596 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19091.39 MB 2025-02-14 08:03:00,613 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 08:03:00,613 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 08:03:00,613 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:03:00,613 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:03:00,613 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18944.34 MB 2025-02-14 08:03:00,613 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19171.09 MB 2025-02-14 08:03:00,613 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.74 MB 2025-02-14 08:03:00,614 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21867.00 MB 2025-02-14 08:03:00,614 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21867.00 MB 2025-02-14 08:03:00,614 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:03:00,614 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19202.37 MB 2025-02-14 08:03:00,616 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 08:03:00,616 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 08:03:00,616 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.59 seconds 2025-02-14 08:03:00,616 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:03:00,616 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13679.46 MB 2025-02-14 08:03:00,616 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19372.16 MB 2025-02-14 08:03:00,616 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5692.70 MB 2025-02-14 08:03:00,616 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58816.72 MB 2025-02-14 08:03:00,616 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21867.00 MB 2025-02-14 08:03:00,616 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -36949.72 MB 2025-02-14 08:03:00,616 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19372.16 MB 2025-02-14 08:03:00,904 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 08:03:00,904 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 08:03:00,904 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-14 08:03:00,904 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:03:00,904 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19372.16 MB 2025-02-14 08:03:00,904 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17706.69 MB 2025-02-14 08:03:00,904 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1665.47 MB 2025-02-14 08:03:00,904 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21867.00 MB 2025-02-14 08:03:00,904 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21867.00 MB 2025-02-14 08:03:00,904 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:03:00,904 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19372.16 MB 2025-02-14 08:03:00,924 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 08:03:00,924 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 08:03:00,931 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 08:03:00,931 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 08:03:00,932 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:03:00,932 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:03:00,932 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17706.69 MB 2025-02-14 08:03:00,932 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26145.72 MB 2025-02-14 08:03:00,932 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 08:03:00,932 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21867.00 MB 2025-02-14 08:03:00,932 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32356.96 MB 2025-02-14 08:03:00,932 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 08:03:00,932 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26145.72 MB 2025-02-14 08:03:01,194 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 08:03:01,197 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:03:01,197 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 08:03:01,199 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:03:01,199 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 08:03:01,207 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 08:03:01,209 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:03:01,209 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 08:03:01,209 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 08:04:33,879 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:04:33,879 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 08:04:33,886 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 08:04:33,892 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:04:33,892 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 907, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 08:04:33,894 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:04:33,894 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 907, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 08:04:47,851 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 08:04:47,852 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 08:04:47,852 - resource_logging.py:150 - __exit__ - DEBUG - Time: 13.95 seconds 2025-02-14 08:04:47,852 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:04:47,852 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19288.83 MB 2025-02-14 08:04:47,852 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22499.57 MB 2025-02-14 08:04:47,852 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3210.74 MB 2025-02-14 08:04:47,852 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44941.97 MB 2025-02-14 08:04:47,852 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30605.84 MB 2025-02-14 08:04:47,852 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14336.13 MB 2025-02-14 08:04:47,852 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31478.11 MB 2025-02-14 08:04:47,907 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 08:04:47,907 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 08:04:47,907 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-14 08:04:47,907 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:04:47,907 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22499.57 MB 2025-02-14 08:04:47,907 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20493.06 MB 2025-02-14 08:04:47,907 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2006.51 MB 2025-02-14 08:04:47,907 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30605.84 MB 2025-02-14 08:04:47,907 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37926.99 MB 2025-02-14 08:04:47,907 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7321.16 MB 2025-02-14 08:04:47,907 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32760.94 MB 2025-02-14 08:04:49,812 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 08:04:49,812 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 08:04:49,812 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.90 seconds 2025-02-14 08:04:49,812 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:04:49,812 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20493.06 MB 2025-02-14 08:04:49,812 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21023.90 MB 2025-02-14 08:04:49,812 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 08:04:49,812 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37926.99 MB 2025-02-14 08:04:49,812 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28810.67 MB 2025-02-14 08:04:49,812 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9116.32 MB 2025-02-14 08:04:49,812 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25003.24 MB 2025-02-14 08:04:49,825 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 08:04:49,825 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 08:04:49,825 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:04:49,825 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:04:49,825 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21023.90 MB 2025-02-14 08:04:49,825 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22913.44 MB 2025-02-14 08:04:49,825 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 08:04:49,825 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28810.67 MB 2025-02-14 08:04:49,825 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28810.67 MB 2025-02-14 08:04:49,825 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:04:49,825 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24330.87 MB 2025-02-14 08:04:50,036 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 08:04:50,036 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 08:04:50,036 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 08:04:50,036 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:04:50,036 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22913.44 MB 2025-02-14 08:04:50,036 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25155.29 MB 2025-02-14 08:04:50,036 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 08:04:50,036 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28810.67 MB 2025-02-14 08:04:50,036 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33057.41 MB 2025-02-14 08:04:50,036 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4246.73 MB 2025-02-14 08:04:50,036 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30699.57 MB 2025-02-14 08:04:50,037 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 08:04:50,037 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 08:04:50,037 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 08:04:50,037 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:04:50,037 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21023.90 MB 2025-02-14 08:04:50,037 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25155.29 MB 2025-02-14 08:04:50,037 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 08:04:50,037 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28810.67 MB 2025-02-14 08:04:50,037 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33057.41 MB 2025-02-14 08:04:50,037 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4246.73 MB 2025-02-14 08:04:50,037 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30699.57 MB 2025-02-14 08:04:50,228 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 08:04:50,228 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 08:04:50,228 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.19 seconds 2025-02-14 08:04:50,228 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:04:50,228 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26688.84 MB 2025-02-14 08:04:50,228 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27455.84 MB 2025-02-14 08:04:50,228 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 08:04:50,228 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33057.41 MB 2025-02-14 08:04:50,228 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33474.74 MB 2025-02-14 08:04:50,228 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 08:04:50,228 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28163.63 MB 2025-02-14 08:04:50,247 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 08:04:50,247 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 08:04:50,247 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:04:50,247 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:04:50,247 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27868.73 MB 2025-02-14 08:04:50,247 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28098.25 MB 2025-02-14 08:04:50,247 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.53 MB 2025-02-14 08:04:50,247 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33474.74 MB 2025-02-14 08:04:50,247 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33474.74 MB 2025-02-14 08:04:50,247 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:04:50,247 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28318.33 MB 2025-02-14 08:04:50,248 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 08:04:50,248 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 08:04:50,248 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.35 seconds 2025-02-14 08:04:50,248 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:04:50,248 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16128.77 MB 2025-02-14 08:04:50,248 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28299.32 MB 2025-02-14 08:04:50,248 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12170.56 MB 2025-02-14 08:04:50,248 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44941.97 MB 2025-02-14 08:04:50,248 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33474.74 MB 2025-02-14 08:04:50,248 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11467.23 MB 2025-02-14 08:04:50,248 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28318.33 MB 2025-02-14 08:04:50,515 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 08:04:50,515 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 08:04:50,515 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 08:04:50,515 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:04:50,515 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28299.32 MB 2025-02-14 08:04:50,515 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21133.16 MB 2025-02-14 08:04:50,515 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7166.17 MB 2025-02-14 08:04:50,515 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33474.74 MB 2025-02-14 08:04:50,515 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33474.74 MB 2025-02-14 08:04:50,515 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:04:50,515 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30810.99 MB 2025-02-14 08:04:50,533 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 08:04:50,533 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 08:04:50,539 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 08:04:50,539 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 08:04:50,539 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:04:50,539 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:04:50,539 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21133.16 MB 2025-02-14 08:04:50,539 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29572.18 MB 2025-02-14 08:04:50,539 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 08:04:50,539 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33474.74 MB 2025-02-14 08:04:50,539 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37671.14 MB 2025-02-14 08:04:50,539 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4196.40 MB 2025-02-14 08:04:50,539 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29572.18 MB 2025-02-14 08:04:50,701 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 08:04:50,703 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:04:50,703 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 08:04:50,703 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:04:50,703 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 08:04:50,708 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 08:04:50,709 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:04:50,709 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 08:04:50,709 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 08:05:05,227 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:05:05,227 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 08:05:05,235 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 08:05:05,241 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:05:05,241 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2152, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 08:05:05,243 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:05:05,243 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2152, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 08:05:38,780 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 08:05:38,780 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 08:05:38,780 - resource_logging.py:150 - __exit__ - DEBUG - Time: 33.53 seconds 2025-02-14 08:05:38,780 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:05:38,780 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27964.19 MB 2025-02-14 08:05:38,780 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35581.04 MB 2025-02-14 08:05:38,780 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7616.86 MB 2025-02-14 08:05:38,780 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50256.15 MB 2025-02-14 08:05:38,780 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41303.41 MB 2025-02-14 08:05:38,780 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8952.74 MB 2025-02-14 08:05:38,780 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44457.63 MB 2025-02-14 08:05:38,908 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 08:05:38,908 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 08:05:38,908 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 08:05:38,908 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:05:38,908 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35581.04 MB 2025-02-14 08:05:38,908 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26965.43 MB 2025-02-14 08:05:38,908 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8615.62 MB 2025-02-14 08:05:38,908 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41303.41 MB 2025-02-14 08:05:38,908 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65601.01 MB 2025-02-14 08:05:38,908 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 24297.60 MB 2025-02-14 08:05:38,908 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 55795.97 MB 2025-02-14 08:05:40,842 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 08:05:40,842 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 08:05:40,842 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 08:05:40,842 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:05:40,842 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26965.43 MB 2025-02-14 08:05:40,842 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27496.27 MB 2025-02-14 08:05:40,842 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 08:05:40,842 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65601.01 MB 2025-02-14 08:05:40,842 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30907.83 MB 2025-02-14 08:05:40,843 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34693.19 MB 2025-02-14 08:05:40,843 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31476.64 MB 2025-02-14 08:05:40,856 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 08:05:40,856 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 08:05:40,856 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:05:40,856 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:05:40,856 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27496.27 MB 2025-02-14 08:05:40,856 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29385.80 MB 2025-02-14 08:05:40,856 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 08:05:40,856 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30907.83 MB 2025-02-14 08:05:40,856 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33738.98 MB 2025-02-14 08:05:40,856 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-14 08:05:40,856 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30803.23 MB 2025-02-14 08:05:41,065 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 08:05:41,065 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 08:05:41,065 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 08:05:41,065 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:05:41,065 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29385.80 MB 2025-02-14 08:05:41,065 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31627.66 MB 2025-02-14 08:05:41,065 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 08:05:41,065 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33738.98 MB 2025-02-14 08:05:41,065 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39401.29 MB 2025-02-14 08:05:41,065 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 08:05:41,066 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37171.94 MB 2025-02-14 08:05:41,066 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 08:05:41,066 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 08:05:41,066 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 08:05:41,066 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:05:41,066 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27496.27 MB 2025-02-14 08:05:41,066 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31627.66 MB 2025-02-14 08:05:41,066 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 08:05:41,066 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30907.83 MB 2025-02-14 08:05:41,066 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39401.29 MB 2025-02-14 08:05:41,066 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8493.47 MB 2025-02-14 08:05:41,066 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37171.94 MB 2025-02-14 08:05:41,232 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 08:05:41,232 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 08:05:41,232 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 08:05:41,232 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:05:41,232 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33161.20 MB 2025-02-14 08:05:41,232 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33928.20 MB 2025-02-14 08:05:41,232 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 08:05:41,232 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39401.29 MB 2025-02-14 08:05:41,232 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39816.53 MB 2025-02-14 08:05:41,232 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 08:05:41,232 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34635.99 MB 2025-02-14 08:05:41,251 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 08:05:41,251 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 08:05:41,251 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:05:41,251 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:05:41,251 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34341.09 MB 2025-02-14 08:05:41,251 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34570.22 MB 2025-02-14 08:05:41,251 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.12 MB 2025-02-14 08:05:41,251 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39816.53 MB 2025-02-14 08:05:41,251 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39816.53 MB 2025-02-14 08:05:41,251 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:05:41,251 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34784.22 MB 2025-02-14 08:05:41,252 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 08:05:41,252 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 08:05:41,252 - resource_logging.py:150 - __exit__ - DEBUG - Time: 36.01 seconds 2025-02-14 08:05:41,252 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:05:41,252 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20466.45 MB 2025-02-14 08:05:41,252 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34770.65 MB 2025-02-14 08:05:41,252 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14304.20 MB 2025-02-14 08:05:41,252 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50256.15 MB 2025-02-14 08:05:41,252 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39816.53 MB 2025-02-14 08:05:41,252 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10439.62 MB 2025-02-14 08:05:41,252 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34784.22 MB 2025-02-14 08:05:41,520 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 08:05:41,520 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 08:05:41,520 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 08:05:41,520 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:05:41,520 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34770.65 MB 2025-02-14 08:05:41,520 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25460.93 MB 2025-02-14 08:05:41,520 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9309.72 MB 2025-02-14 08:05:41,520 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39816.53 MB 2025-02-14 08:05:41,520 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39816.53 MB 2025-02-14 08:05:41,520 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:05:41,520 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37274.33 MB 2025-02-14 08:05:41,538 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8136, cut from 8138 2025-02-14 08:05:41,538 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 08:05:41,544 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 08:05:41,544 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 08:05:41,544 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:05:41,544 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:05:41,544 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25460.93 MB 2025-02-14 08:05:41,544 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33873.36 MB 2025-02-14 08:05:41,544 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8412.43 MB 2025-02-14 08:05:41,544 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39816.53 MB 2025-02-14 08:05:41,544 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48179.97 MB 2025-02-14 08:05:41,544 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8363.44 MB 2025-02-14 08:05:41,544 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33873.36 MB 2025-02-14 08:05:41,706 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7928] 2025-02-14 08:05:41,707 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:05:41,707 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 08:05:41,708 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:05:41,708 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 08:05:41,713 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 08:05:41,714 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:05:41,714 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 08:05:41,714 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 08:07:39,118 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:07:39,118 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 08:07:39,123 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 08:07:39,127 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:07:39,127 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 225, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 08:07:39,129 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:07:39,129 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 225, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 08:07:42,592 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 08:07:42,592 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 08:07:42,592 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.46 seconds 2025-02-14 08:07:42,592 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:07:42,593 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14537.16 MB 2025-02-14 08:07:42,593 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15333.42 MB 2025-02-14 08:07:42,593 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 796.26 MB 2025-02-14 08:07:42,593 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56543.41 MB 2025-02-14 08:07:42,593 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21101.54 MB 2025-02-14 08:07:42,593 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35441.87 MB 2025-02-14 08:07:42,593 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24235.02 MB 2025-02-14 08:07:42,607 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 08:07:42,607 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 08:07:42,607 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:07:42,607 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:07:42,607 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15333.42 MB 2025-02-14 08:07:42,607 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15311.87 MB 2025-02-14 08:07:42,607 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -21.55 MB 2025-02-14 08:07:42,607 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21101.54 MB 2025-02-14 08:07:42,607 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21101.54 MB 2025-02-14 08:07:42,607 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:07:42,607 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17683.64 MB 2025-02-14 08:07:43,398 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 08:07:43,399 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 08:07:43,399 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.79 seconds 2025-02-14 08:07:43,399 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:07:43,399 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15311.87 MB 2025-02-14 08:07:43,399 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15533.50 MB 2025-02-14 08:07:43,399 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 221.63 MB 2025-02-14 08:07:43,399 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21101.54 MB 2025-02-14 08:07:43,399 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21101.54 MB 2025-02-14 08:07:43,399 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:07:43,399 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19482.31 MB 2025-02-14 08:07:43,407 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 08:07:43,407 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 08:07:43,407 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 08:07:43,407 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:07:43,407 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15533.43 MB 2025-02-14 08:07:43,407 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16322.12 MB 2025-02-14 08:07:43,407 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 788.69 MB 2025-02-14 08:07:43,407 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21101.54 MB 2025-02-14 08:07:43,407 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21101.54 MB 2025-02-14 08:07:43,407 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:07:43,407 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16913.90 MB 2025-02-14 08:07:43,498 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 08:07:43,498 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 08:07:43,498 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 08:07:43,498 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:07:43,498 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16322.12 MB 2025-02-14 08:07:43,498 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17258.13 MB 2025-02-14 08:07:43,498 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 936.01 MB 2025-02-14 08:07:43,498 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21101.54 MB 2025-02-14 08:07:43,498 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21101.54 MB 2025-02-14 08:07:43,498 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:07:43,498 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19572.83 MB 2025-02-14 08:07:43,499 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 08:07:43,499 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 08:07:43,499 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 08:07:43,499 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:07:43,499 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15533.43 MB 2025-02-14 08:07:43,499 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17258.13 MB 2025-02-14 08:07:43,499 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1724.70 MB 2025-02-14 08:07:43,499 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21101.54 MB 2025-02-14 08:07:43,499 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21101.54 MB 2025-02-14 08:07:43,499 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:07:43,499 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19572.83 MB 2025-02-14 08:07:43,570 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 08:07:43,570 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 08:07:43,570 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 08:07:43,570 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:07:43,570 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17898.39 MB 2025-02-14 08:07:43,570 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18218.61 MB 2025-02-14 08:07:43,570 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 320.22 MB 2025-02-14 08:07:43,570 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21101.54 MB 2025-02-14 08:07:43,570 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21273.51 MB 2025-02-14 08:07:43,570 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 171.97 MB 2025-02-14 08:07:43,570 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18521.51 MB 2025-02-14 08:07:43,579 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 08:07:43,579 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 08:07:43,580 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:07:43,580 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:07:43,580 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18391.00 MB 2025-02-14 08:07:43,580 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18617.26 MB 2025-02-14 08:07:43,580 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.26 MB 2025-02-14 08:07:43,580 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21273.51 MB 2025-02-14 08:07:43,580 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21273.51 MB 2025-02-14 08:07:43,580 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:07:43,580 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18641.28 MB 2025-02-14 08:07:43,581 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 08:07:43,581 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 08:07:43,581 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.45 seconds 2025-02-14 08:07:43,581 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:07:43,581 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13753.24 MB 2025-02-14 08:07:43,581 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18818.26 MB 2025-02-14 08:07:43,581 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5065.02 MB 2025-02-14 08:07:43,581 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56543.41 MB 2025-02-14 08:07:43,581 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21273.51 MB 2025-02-14 08:07:43,581 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35269.90 MB 2025-02-14 08:07:43,581 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18818.26 MB 2025-02-14 08:07:43,846 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 08:07:43,846 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 08:07:43,846 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 08:07:43,846 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:07:43,846 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18818.26 MB 2025-02-14 08:07:43,846 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17656.89 MB 2025-02-14 08:07:43,846 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1161.37 MB 2025-02-14 08:07:43,846 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21273.51 MB 2025-02-14 08:07:43,846 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21273.51 MB 2025-02-14 08:07:43,846 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:07:43,846 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19420.84 MB 2025-02-14 08:07:43,864 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8159, cut from 8161 2025-02-14 08:07:43,864 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 08:07:43,870 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 08:07:43,870 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 08:07:43,871 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:07:43,871 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:07:43,871 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17656.89 MB 2025-02-14 08:07:43,871 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26092.48 MB 2025-02-14 08:07:43,871 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8435.59 MB 2025-02-14 08:07:43,871 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21273.51 MB 2025-02-14 08:07:43,871 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31759.27 MB 2025-02-14 08:07:43,871 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10485.76 MB 2025-02-14 08:07:43,871 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26092.48 MB 2025-02-14 08:07:44,023 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7951] 2025-02-14 08:07:44,025 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:07:44,025 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 08:07:44,026 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:07:44,026 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 08:07:44,030 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 08:07:44,031 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:07:44,031 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 08:07:44,031 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 08:07:52,690 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:07:52,690 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 08:07:52,695 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 08:07:52,698 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:07:52,698 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2463, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 08:07:52,699 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:07:52,699 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2463, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 08:08:30,984 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 08:08:30,985 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 08:08:30,985 - resource_logging.py:150 - __exit__ - DEBUG - Time: 38.27 seconds 2025-02-14 08:08:30,985 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:08:30,985 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30132.45 MB 2025-02-14 08:08:30,985 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38848.87 MB 2025-02-14 08:08:30,985 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8716.42 MB 2025-02-14 08:08:30,985 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57315.16 MB 2025-02-14 08:08:30,985 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42792.39 MB 2025-02-14 08:08:30,985 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14522.78 MB 2025-02-14 08:08:30,985 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47758.36 MB 2025-02-14 08:08:31,149 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 08:08:31,149 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 08:08:31,149 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 08:08:31,149 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:08:31,149 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38848.87 MB 2025-02-14 08:08:31,149 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28583.39 MB 2025-02-14 08:08:31,149 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10265.48 MB 2025-02-14 08:08:31,149 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42792.39 MB 2025-02-14 08:08:31,149 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 76172.75 MB 2025-02-14 08:08:31,149 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 33380.37 MB 2025-02-14 08:08:31,149 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 64421.78 MB 2025-02-14 08:08:33,133 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 08:08:33,133 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 08:08:33,133 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.98 seconds 2025-02-14 08:08:33,133 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:08:33,133 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28583.39 MB 2025-02-14 08:08:33,133 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29114.23 MB 2025-02-14 08:08:33,133 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 08:08:33,133 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 76172.75 MB 2025-02-14 08:08:33,134 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31130.12 MB 2025-02-14 08:08:33,134 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -45042.63 MB 2025-02-14 08:08:33,134 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33094.60 MB 2025-02-14 08:08:33,148 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 08:08:33,148 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 08:08:33,148 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:08:33,148 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:08:33,148 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29114.23 MB 2025-02-14 08:08:33,148 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31003.44 MB 2025-02-14 08:08:33,148 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.21 MB 2025-02-14 08:08:33,148 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31130.12 MB 2025-02-14 08:08:33,148 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34433.14 MB 2025-02-14 08:08:33,148 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 08:08:33,148 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32420.87 MB 2025-02-14 08:08:33,359 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 08:08:33,359 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 08:08:33,359 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 08:08:33,359 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:08:33,359 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31003.44 MB 2025-02-14 08:08:33,359 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33245.29 MB 2025-02-14 08:08:33,359 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 08:08:33,359 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34433.14 MB 2025-02-14 08:08:33,359 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41039.17 MB 2025-02-14 08:08:33,359 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 08:08:33,359 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38789.57 MB 2025-02-14 08:08:33,360 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 08:08:33,360 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 08:08:33,360 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 08:08:33,360 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:08:33,360 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29114.23 MB 2025-02-14 08:08:33,360 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33245.29 MB 2025-02-14 08:08:33,360 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.06 MB 2025-02-14 08:08:33,360 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31130.12 MB 2025-02-14 08:08:33,360 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41039.17 MB 2025-02-14 08:08:33,360 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9909.04 MB 2025-02-14 08:08:33,360 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38789.57 MB 2025-02-14 08:08:33,527 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 08:08:33,527 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 08:08:33,527 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 08:08:33,528 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:08:33,528 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34778.83 MB 2025-02-14 08:08:33,528 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35545.84 MB 2025-02-14 08:08:33,528 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 08:08:33,528 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41039.17 MB 2025-02-14 08:08:33,528 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41456.50 MB 2025-02-14 08:08:33,528 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 08:08:33,528 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36253.62 MB 2025-02-14 08:08:33,547 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 08:08:33,547 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 08:08:33,547 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:08:33,547 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:08:33,547 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35958.73 MB 2025-02-14 08:08:33,547 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36187.83 MB 2025-02-14 08:08:33,547 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.11 MB 2025-02-14 08:08:33,547 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41456.50 MB 2025-02-14 08:08:33,547 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41456.50 MB 2025-02-14 08:08:33,547 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:08:33,547 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36412.02 MB 2025-02-14 08:08:33,548 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 08:08:33,548 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 08:08:33,548 - resource_logging.py:150 - __exit__ - DEBUG - Time: 40.85 seconds 2025-02-14 08:08:33,548 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:08:33,548 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21550.58 MB 2025-02-14 08:08:33,548 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36388.86 MB 2025-02-14 08:08:33,548 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14838.28 MB 2025-02-14 08:08:33,548 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48731.52 MB 2025-02-14 08:08:33,548 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41456.50 MB 2025-02-14 08:08:33,548 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7275.02 MB 2025-02-14 08:08:33,548 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36412.02 MB 2025-02-14 08:08:33,821 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 08:08:33,821 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 08:08:33,821 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 08:08:33,821 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:08:33,821 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36388.86 MB 2025-02-14 08:08:33,822 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26554.21 MB 2025-02-14 08:08:33,822 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9834.65 MB 2025-02-14 08:08:33,822 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41456.50 MB 2025-02-14 08:08:33,822 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41456.50 MB 2025-02-14 08:08:33,822 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:08:33,822 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38899.91 MB 2025-02-14 08:08:33,840 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8160, cut from 8162 2025-02-14 08:08:33,840 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 08:08:33,846 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 08:08:33,846 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 08:08:33,846 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:08:33,846 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:08:33,846 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26554.21 MB 2025-02-14 08:08:33,846 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34991.68 MB 2025-02-14 08:08:33,846 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8437.47 MB 2025-02-14 08:08:33,846 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41456.50 MB 2025-02-14 08:08:33,846 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45650.80 MB 2025-02-14 08:08:33,846 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4194.30 MB 2025-02-14 08:08:33,846 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34991.68 MB 2025-02-14 08:08:34,015 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7952] 2025-02-14 08:08:34,016 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:08:34,016 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 08:08:34,017 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:08:34,017 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 08:08:34,022 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 08:08:34,023 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:08:34,023 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 08:08:34,023 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 08:08:43,923 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:08:43,924 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 08:08:43,928 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 08:08:43,932 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:08:43,932 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 165, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 08:08:43,934 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:08:43,934 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 165, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 08:08:46,533 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 08:08:46,533 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 08:08:46,533 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.59 seconds 2025-02-14 08:08:46,533 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:08:46,533 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14118.45 MB 2025-02-14 08:08:46,533 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14702.38 MB 2025-02-14 08:08:46,533 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 583.93 MB 2025-02-14 08:08:46,533 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54039.41 MB 2025-02-14 08:08:46,533 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 16909.34 MB 2025-02-14 08:08:46,533 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -37130.08 MB 2025-02-14 08:08:46,533 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23590.63 MB 2025-02-14 08:08:46,546 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 08:08:46,546 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 08:08:46,546 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:08:46,546 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:08:46,546 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14702.38 MB 2025-02-14 08:08:46,546 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14858.88 MB 2025-02-14 08:08:46,546 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 156.50 MB 2025-02-14 08:08:46,546 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 16909.34 MB 2025-02-14 08:08:46,546 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17953.72 MB 2025-02-14 08:08:46,546 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1044.38 MB 2025-02-14 08:08:46,546 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16785.80 MB 2025-02-14 08:08:47,263 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 08:08:47,263 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 08:08:47,263 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.72 seconds 2025-02-14 08:08:47,263 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:08:47,263 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14858.88 MB 2025-02-14 08:08:47,263 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15053.96 MB 2025-02-14 08:08:47,263 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 195.08 MB 2025-02-14 08:08:47,263 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17953.72 MB 2025-02-14 08:08:47,263 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17834.18 MB 2025-02-14 08:08:47,263 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -119.54 MB 2025-02-14 08:08:47,263 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19030.35 MB 2025-02-14 08:08:47,270 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 08:08:47,270 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 08:08:47,270 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 08:08:47,270 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:08:47,270 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15053.89 MB 2025-02-14 08:08:47,270 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15748.13 MB 2025-02-14 08:08:47,270 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 694.24 MB 2025-02-14 08:08:47,270 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17834.18 MB 2025-02-14 08:08:47,270 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17834.18 MB 2025-02-14 08:08:47,270 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:08:47,270 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16269.04 MB 2025-02-14 08:08:47,352 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 08:08:47,352 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 08:08:47,352 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 08:08:47,352 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:08:47,352 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15748.13 MB 2025-02-14 08:08:47,352 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16572.05 MB 2025-02-14 08:08:47,352 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 823.92 MB 2025-02-14 08:08:47,352 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17834.18 MB 2025-02-14 08:08:47,352 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19922.94 MB 2025-02-14 08:08:47,352 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2088.76 MB 2025-02-14 08:08:47,352 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18610.85 MB 2025-02-14 08:08:47,352 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 08:08:47,352 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 08:08:47,352 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 08:08:47,352 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:08:47,352 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15053.89 MB 2025-02-14 08:08:47,353 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16572.05 MB 2025-02-14 08:08:47,353 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1518.16 MB 2025-02-14 08:08:47,353 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17834.18 MB 2025-02-14 08:08:47,353 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19922.94 MB 2025-02-14 08:08:47,353 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2088.76 MB 2025-02-14 08:08:47,353 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18610.85 MB 2025-02-14 08:08:47,431 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 08:08:47,431 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 08:08:47,431 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 08:08:47,431 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:08:47,431 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17135.63 MB 2025-02-14 08:08:47,431 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17418.81 MB 2025-02-14 08:08:47,431 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 283.18 MB 2025-02-14 08:08:47,431 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19922.94 MB 2025-02-14 08:08:47,431 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20071.84 MB 2025-02-14 08:08:47,431 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 148.90 MB 2025-02-14 08:08:47,431 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17689.42 MB 2025-02-14 08:08:47,440 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 08:08:47,440 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 08:08:47,440 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:08:47,440 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:08:47,440 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17570.56 MB 2025-02-14 08:08:47,440 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17779.88 MB 2025-02-14 08:08:47,440 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 209.32 MB 2025-02-14 08:08:47,440 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20071.84 MB 2025-02-14 08:08:47,440 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20071.84 MB 2025-02-14 08:08:47,440 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:08:47,440 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17791.31 MB 2025-02-14 08:08:47,441 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 08:08:47,441 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 08:08:47,441 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.50 seconds 2025-02-14 08:08:47,441 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:08:47,441 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13543.58 MB 2025-02-14 08:08:47,441 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17980.80 MB 2025-02-14 08:08:47,441 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4437.22 MB 2025-02-14 08:08:47,441 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54039.41 MB 2025-02-14 08:08:47,441 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20071.84 MB 2025-02-14 08:08:47,441 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33967.57 MB 2025-02-14 08:08:47,441 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17980.80 MB 2025-02-14 08:08:47,710 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 08:08:47,710 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 08:08:47,710 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 08:08:47,710 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:08:47,710 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17980.80 MB 2025-02-14 08:08:47,710 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17352.88 MB 2025-02-14 08:08:47,710 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -627.92 MB 2025-02-14 08:08:47,710 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20071.84 MB 2025-02-14 08:08:47,710 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20071.84 MB 2025-02-14 08:08:47,710 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:08:47,710 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19085.13 MB 2025-02-14 08:08:47,729 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8156, cut from 8158 2025-02-14 08:08:47,729 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2,'] 2025-02-14 08:08:47,735 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 08:08:47,735 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 08:08:47,735 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:08:47,735 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:08:47,735 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17352.88 MB 2025-02-14 08:08:47,735 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25786.18 MB 2025-02-14 08:08:47,735 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8433.30 MB 2025-02-14 08:08:47,735 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20071.84 MB 2025-02-14 08:08:47,735 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30553.41 MB 2025-02-14 08:08:47,735 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10481.57 MB 2025-02-14 08:08:47,735 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25786.18 MB 2025-02-14 08:08:47,897 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7948] 2025-02-14 08:08:47,899 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:08:47,899 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 08:08:47,900 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:08:47,900 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 08:08:47,905 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 08:08:47,906 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:08:47,906 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 08:08:47,906 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2,'] 2025-02-14 08:08:57,187 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:08:57,187 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 08:08:57,192 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 08:08:57,195 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:08:57,195 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 172, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 08:08:57,196 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:08:57,196 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 172, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 08:08:59,879 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 08:08:59,879 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 08:08:59,879 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.68 seconds 2025-02-14 08:08:59,879 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:08:59,879 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14167.23 MB 2025-02-14 08:08:59,879 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14775.93 MB 2025-02-14 08:08:59,879 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 608.70 MB 2025-02-14 08:08:59,879 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38937.82 MB 2025-02-14 08:08:59,879 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18125.68 MB 2025-02-14 08:08:59,879 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20812.14 MB 2025-02-14 08:08:59,879 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23639.41 MB 2025-02-14 08:08:59,892 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 08:08:59,892 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 08:08:59,892 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:08:59,892 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:08:59,892 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14775.93 MB 2025-02-14 08:08:59,892 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15070.84 MB 2025-02-14 08:08:59,892 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 294.91 MB 2025-02-14 08:08:59,892 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18125.68 MB 2025-02-14 08:08:59,892 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18735.96 MB 2025-02-14 08:08:59,892 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 610.27 MB 2025-02-14 08:08:59,892 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17234.38 MB 2025-02-14 08:09:00,724 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 08:09:00,724 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 08:09:00,724 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.83 seconds 2025-02-14 08:09:00,724 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:09:00,724 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15070.84 MB 2025-02-14 08:09:00,724 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15299.10 MB 2025-02-14 08:09:00,724 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.26 MB 2025-02-14 08:09:00,724 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18735.96 MB 2025-02-14 08:09:00,724 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18387.83 MB 2025-02-14 08:09:00,724 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -348.13 MB 2025-02-14 08:09:00,724 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19242.32 MB 2025-02-14 08:09:00,733 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 08:09:00,733 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 08:09:00,733 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:09:00,733 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:09:00,733 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15299.04 MB 2025-02-14 08:09:00,733 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16111.34 MB 2025-02-14 08:09:00,733 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 812.30 MB 2025-02-14 08:09:00,733 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18387.83 MB 2025-02-14 08:09:00,733 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18387.83 MB 2025-02-14 08:09:00,733 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:09:00,733 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16720.84 MB 2025-02-14 08:09:00,828 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 08:09:00,828 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 08:09:00,828 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 08:09:00,828 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:09:00,828 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16111.34 MB 2025-02-14 08:09:00,828 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17075.38 MB 2025-02-14 08:09:00,828 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 964.04 MB 2025-02-14 08:09:00,828 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18387.83 MB 2025-02-14 08:09:00,828 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20422.07 MB 2025-02-14 08:09:00,828 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2034.24 MB 2025-02-14 08:09:00,828 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19463.57 MB 2025-02-14 08:09:00,828 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 08:09:00,829 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 08:09:00,829 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 08:09:00,829 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:09:00,829 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15299.04 MB 2025-02-14 08:09:00,829 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17075.38 MB 2025-02-14 08:09:00,829 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1776.34 MB 2025-02-14 08:09:00,829 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18387.83 MB 2025-02-14 08:09:00,829 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20422.07 MB 2025-02-14 08:09:00,829 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2034.24 MB 2025-02-14 08:09:00,829 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19463.57 MB 2025-02-14 08:09:00,905 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 08:09:00,905 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 08:09:00,905 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 08:09:00,905 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:09:00,905 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17734.80 MB 2025-02-14 08:09:00,905 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18064.61 MB 2025-02-14 08:09:00,905 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 329.81 MB 2025-02-14 08:09:00,905 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20422.07 MB 2025-02-14 08:09:00,905 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20600.32 MB 2025-02-14 08:09:00,905 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 178.26 MB 2025-02-14 08:09:00,905 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18377.27 MB 2025-02-14 08:09:00,915 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 08:09:00,915 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 08:09:00,915 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:09:00,915 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:09:00,915 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18242.16 MB 2025-02-14 08:09:00,915 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18452.35 MB 2025-02-14 08:09:00,915 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 210.19 MB 2025-02-14 08:09:00,915 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20600.32 MB 2025-02-14 08:09:00,915 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20600.32 MB 2025-02-14 08:09:00,915 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:09:00,915 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18480.81 MB 2025-02-14 08:09:00,916 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 08:09:00,916 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 08:09:00,916 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.72 seconds 2025-02-14 08:09:00,916 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:09:00,916 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13567.97 MB 2025-02-14 08:09:00,916 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18653.43 MB 2025-02-14 08:09:00,916 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5085.46 MB 2025-02-14 08:09:00,916 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38937.82 MB 2025-02-14 08:09:00,916 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20600.32 MB 2025-02-14 08:09:00,916 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18337.50 MB 2025-02-14 08:09:00,916 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18653.43 MB 2025-02-14 08:09:01,185 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 08:09:01,185 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 08:09:01,186 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 08:09:01,186 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:09:01,186 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18653.43 MB 2025-02-14 08:09:01,186 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17495.83 MB 2025-02-14 08:09:01,186 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1157.59 MB 2025-02-14 08:09:01,186 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20600.32 MB 2025-02-14 08:09:01,186 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20600.32 MB 2025-02-14 08:09:01,186 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:09:01,186 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18887.85 MB 2025-02-14 08:09:01,204 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 08:09:01,204 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 08:09:01,210 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 08:09:01,210 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 08:09:01,210 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:09:01,210 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:09:01,210 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17495.83 MB 2025-02-14 08:09:01,210 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25934.86 MB 2025-02-14 08:09:01,210 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 08:09:01,210 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20600.32 MB 2025-02-14 08:09:01,210 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31090.28 MB 2025-02-14 08:09:01,210 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 08:09:01,210 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25934.86 MB 2025-02-14 08:09:01,380 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 08:09:01,381 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:09:01,381 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 08:09:01,382 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:09:01,382 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 08:09:01,387 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 08:09:01,388 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:09:01,388 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 08:09:01,388 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 08:09:23,637 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:09:23,637 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 08:09:23,642 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 08:09:23,645 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:09:23,645 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 181, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 08:09:23,646 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:09:23,646 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 181, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 08:09:26,455 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 08:09:26,455 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 08:09:26,455 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.81 seconds 2025-02-14 08:09:26,456 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:09:26,456 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14229.94 MB 2025-02-14 08:09:26,456 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14870.49 MB 2025-02-14 08:09:26,456 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 640.55 MB 2025-02-14 08:09:26,456 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43675.29 MB 2025-02-14 08:09:26,456 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17664.31 MB 2025-02-14 08:09:26,456 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26010.98 MB 2025-02-14 08:09:26,456 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23702.12 MB 2025-02-14 08:09:26,470 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 08:09:26,470 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 08:09:26,470 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:09:26,470 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:09:26,471 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14870.49 MB 2025-02-14 08:09:26,471 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15061.97 MB 2025-02-14 08:09:26,471 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 191.48 MB 2025-02-14 08:09:26,471 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17664.31 MB 2025-02-14 08:09:26,471 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18245.22 MB 2025-02-14 08:09:26,471 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 580.91 MB 2025-02-14 08:09:26,471 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17192.33 MB 2025-02-14 08:09:27,265 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 08:09:27,265 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 08:09:27,265 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.79 seconds 2025-02-14 08:09:27,265 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:09:27,265 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15061.97 MB 2025-02-14 08:09:27,265 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15279.61 MB 2025-02-14 08:09:27,265 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 217.65 MB 2025-02-14 08:09:27,265 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18245.22 MB 2025-02-14 08:09:27,265 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18245.22 MB 2025-02-14 08:09:27,265 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:09:27,265 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19233.44 MB 2025-02-14 08:09:27,273 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 08:09:27,273 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 08:09:27,273 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 08:09:27,273 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:09:27,273 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15279.55 MB 2025-02-14 08:09:27,273 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16054.07 MB 2025-02-14 08:09:27,273 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 774.52 MB 2025-02-14 08:09:27,273 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18245.22 MB 2025-02-14 08:09:27,273 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18245.22 MB 2025-02-14 08:09:27,273 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:09:27,273 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16635.22 MB 2025-02-14 08:09:27,363 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 08:09:27,363 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 08:09:27,363 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 08:09:27,363 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:09:27,363 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16054.07 MB 2025-02-14 08:09:27,363 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16973.27 MB 2025-02-14 08:09:27,363 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 919.20 MB 2025-02-14 08:09:27,363 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18245.22 MB 2025-02-14 08:09:27,363 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20380.12 MB 2025-02-14 08:09:27,363 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2134.90 MB 2025-02-14 08:09:27,363 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19250.58 MB 2025-02-14 08:09:27,364 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 08:09:27,364 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 08:09:27,364 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 08:09:27,364 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:09:27,364 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15279.55 MB 2025-02-14 08:09:27,364 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16973.27 MB 2025-02-14 08:09:27,364 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1693.72 MB 2025-02-14 08:09:27,364 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18245.22 MB 2025-02-14 08:09:27,364 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20380.12 MB 2025-02-14 08:09:27,364 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2134.90 MB 2025-02-14 08:09:27,364 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19250.58 MB 2025-02-14 08:09:27,433 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 08:09:27,433 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 08:09:27,433 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 08:09:27,433 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:09:27,433 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17602.02 MB 2025-02-14 08:09:27,433 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17916.49 MB 2025-02-14 08:09:27,433 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 314.47 MB 2025-02-14 08:09:27,433 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20380.12 MB 2025-02-14 08:09:27,433 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20547.90 MB 2025-02-14 08:09:27,433 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 167.77 MB 2025-02-14 08:09:27,433 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18215.20 MB 2025-02-14 08:09:27,442 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 08:09:27,442 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 08:09:27,442 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:09:27,442 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:09:27,443 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18085.78 MB 2025-02-14 08:09:27,443 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18299.31 MB 2025-02-14 08:09:27,443 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 213.53 MB 2025-02-14 08:09:27,443 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20547.90 MB 2025-02-14 08:09:27,443 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20547.90 MB 2025-02-14 08:09:27,443 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:09:27,443 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18327.97 MB 2025-02-14 08:09:27,444 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 08:09:27,444 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 08:09:27,444 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.80 seconds 2025-02-14 08:09:27,444 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:09:27,444 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13599.32 MB 2025-02-14 08:09:27,444 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18500.36 MB 2025-02-14 08:09:27,444 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4901.04 MB 2025-02-14 08:09:27,444 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43675.29 MB 2025-02-14 08:09:27,444 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20547.90 MB 2025-02-14 08:09:27,444 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23127.39 MB 2025-02-14 08:09:27,444 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18500.36 MB 2025-02-14 08:09:27,711 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 08:09:27,711 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 08:09:27,711 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 08:09:27,711 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:09:27,711 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18500.36 MB 2025-02-14 08:09:27,711 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17489.06 MB 2025-02-14 08:09:27,711 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1011.30 MB 2025-02-14 08:09:27,711 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20547.90 MB 2025-02-14 08:09:27,711 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20547.90 MB 2025-02-14 08:09:27,711 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:09:27,711 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19203.54 MB 2025-02-14 08:09:27,729 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8161, cut from 8163 2025-02-14 08:09:27,729 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 08:09:27,735 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 08:09:27,735 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 08:09:27,735 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:09:27,735 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:09:27,735 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17489.06 MB 2025-02-14 08:09:27,735 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25927.89 MB 2025-02-14 08:09:27,735 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8438.84 MB 2025-02-14 08:09:27,735 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20547.90 MB 2025-02-14 08:09:27,735 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31033.66 MB 2025-02-14 08:09:27,735 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10485.76 MB 2025-02-14 08:09:27,735 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25927.89 MB 2025-02-14 08:09:27,897 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7953] 2025-02-14 08:09:27,899 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:09:27,899 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 08:09:27,900 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:09:27,900 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 08:09:27,904 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 08:09:27,905 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:09:27,905 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 08:09:27,906 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 08:09:36,394 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:09:36,394 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 08:09:36,399 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 08:09:36,403 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:09:36,403 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 473, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 08:09:36,404 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:09:36,404 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 473, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 08:09:43,752 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 08:09:43,752 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 08:09:43,752 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.34 seconds 2025-02-14 08:09:43,752 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:09:43,752 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16264.65 MB 2025-02-14 08:09:43,752 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17938.57 MB 2025-02-14 08:09:43,752 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1673.92 MB 2025-02-14 08:09:43,752 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39422.26 MB 2025-02-14 08:09:43,752 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23200.79 MB 2025-02-14 08:09:43,752 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16221.47 MB 2025-02-14 08:09:43,752 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26868.48 MB 2025-02-14 08:09:43,785 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 08:09:43,786 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 08:09:43,786 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 08:09:43,786 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:09:43,786 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17938.57 MB 2025-02-14 08:09:43,786 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18237.88 MB 2025-02-14 08:09:43,786 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 299.31 MB 2025-02-14 08:09:43,786 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23200.79 MB 2025-02-14 08:09:43,786 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28349.30 MB 2025-02-14 08:09:43,786 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5148.51 MB 2025-02-14 08:09:43,786 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25384.71 MB 2025-02-14 08:09:45,704 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 08:09:45,705 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 08:09:45,705 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 08:09:45,705 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:09:45,705 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18237.88 MB 2025-02-14 08:09:45,705 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18768.72 MB 2025-02-14 08:09:45,705 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 08:09:45,705 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28349.30 MB 2025-02-14 08:09:45,705 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25325.21 MB 2025-02-14 08:09:45,705 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3024.09 MB 2025-02-14 08:09:45,705 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22748.05 MB 2025-02-14 08:09:45,718 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 08:09:45,719 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 08:09:45,719 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:09:45,719 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:09:45,719 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18768.72 MB 2025-02-14 08:09:45,719 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20658.25 MB 2025-02-14 08:09:45,719 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 08:09:45,719 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25325.21 MB 2025-02-14 08:09:45,719 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25325.21 MB 2025-02-14 08:09:45,719 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:09:45,719 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22075.68 MB 2025-02-14 08:09:45,932 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 08:09:45,932 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 08:09:45,932 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 08:09:45,932 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:09:45,932 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20658.25 MB 2025-02-14 08:09:45,932 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22900.11 MB 2025-02-14 08:09:45,932 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 08:09:45,932 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25325.21 MB 2025-02-14 08:09:45,932 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30987.52 MB 2025-02-14 08:09:45,932 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 08:09:45,932 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28444.39 MB 2025-02-14 08:09:45,932 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 08:09:45,932 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 08:09:45,932 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 08:09:45,932 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:09:45,932 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18768.72 MB 2025-02-14 08:09:45,932 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22900.11 MB 2025-02-14 08:09:45,933 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 08:09:45,933 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25325.21 MB 2025-02-14 08:09:45,933 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30987.52 MB 2025-02-14 08:09:45,933 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 08:09:45,933 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28444.39 MB 2025-02-14 08:09:46,099 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 08:09:46,099 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 08:09:46,099 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 08:09:46,099 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:09:46,099 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24433.65 MB 2025-02-14 08:09:46,099 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25200.65 MB 2025-02-14 08:09:46,099 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 08:09:46,099 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30987.52 MB 2025-02-14 08:09:46,099 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31402.75 MB 2025-02-14 08:09:46,099 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 08:09:46,099 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25908.44 MB 2025-02-14 08:09:46,118 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 08:09:46,118 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 08:09:46,118 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:09:46,118 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:09:46,118 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25613.54 MB 2025-02-14 08:09:46,118 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25847.05 MB 2025-02-14 08:09:46,118 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 233.51 MB 2025-02-14 08:09:46,118 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31402.75 MB 2025-02-14 08:09:46,118 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31402.75 MB 2025-02-14 08:09:46,118 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:09:46,118 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26028.90 MB 2025-02-14 08:09:46,119 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 08:09:46,119 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 08:09:46,119 - resource_logging.py:150 - __exit__ - DEBUG - Time: 9.71 seconds 2025-02-14 08:09:46,119 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:09:46,119 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14616.68 MB 2025-02-14 08:09:46,119 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26048.12 MB 2025-02-14 08:09:46,119 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11431.45 MB 2025-02-14 08:09:46,119 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39422.26 MB 2025-02-14 08:09:46,119 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31402.75 MB 2025-02-14 08:09:46,120 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8019.51 MB 2025-02-14 08:09:46,120 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26048.12 MB 2025-02-14 08:09:46,387 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 08:09:46,387 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 08:09:46,387 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 08:09:46,387 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:09:46,387 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26048.12 MB 2025-02-14 08:09:46,387 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19621.07 MB 2025-02-14 08:09:46,387 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6427.06 MB 2025-02-14 08:09:46,387 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31402.75 MB 2025-02-14 08:09:46,387 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31402.75 MB 2025-02-14 08:09:46,387 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:09:46,387 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28559.79 MB 2025-02-14 08:09:46,405 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 08:09:46,406 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 08:09:46,412 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 08:09:46,412 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 08:09:46,412 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:09:46,412 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:09:46,412 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19621.07 MB 2025-02-14 08:09:46,412 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28060.09 MB 2025-02-14 08:09:46,412 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 08:09:46,412 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31402.75 MB 2025-02-14 08:09:46,412 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39793.46 MB 2025-02-14 08:09:46,412 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 08:09:46,412 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28060.09 MB 2025-02-14 08:09:46,575 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 08:09:46,576 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:09:46,576 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 08:09:46,577 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:09:46,577 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 08:09:46,582 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 08:09:46,583 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:09:46,583 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 08:09:46,583 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 08:10:44,794 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:10:44,795 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 08:10:44,799 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 08:10:44,803 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:10:44,803 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 152, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 08:10:44,804 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:10:44,804 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 152, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 08:10:47,217 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 08:10:47,217 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 08:10:47,217 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.41 seconds 2025-02-14 08:10:47,217 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:10:47,217 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14027.87 MB 2025-02-14 08:10:47,217 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14565.79 MB 2025-02-14 08:10:47,217 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 537.92 MB 2025-02-14 08:10:47,217 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52378.47 MB 2025-02-14 08:10:47,217 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17853.05 MB 2025-02-14 08:10:47,217 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34525.41 MB 2025-02-14 08:10:47,217 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23500.04 MB 2025-02-14 08:10:47,232 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 08:10:47,232 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 08:10:47,232 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:10:47,232 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:10:47,232 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14565.79 MB 2025-02-14 08:10:47,232 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14784.27 MB 2025-02-14 08:10:47,232 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 218.48 MB 2025-02-14 08:10:47,233 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17853.05 MB 2025-02-14 08:10:47,233 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18371.05 MB 2025-02-14 08:10:47,233 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 518.00 MB 2025-02-14 08:10:47,233 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16644.89 MB 2025-02-14 08:10:47,962 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 08:10:47,962 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 08:10:47,962 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.73 seconds 2025-02-14 08:10:47,962 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:10:47,962 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14784.27 MB 2025-02-14 08:10:47,962 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14978.03 MB 2025-02-14 08:10:47,962 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 193.76 MB 2025-02-14 08:10:47,962 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18371.05 MB 2025-02-14 08:10:47,962 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17899.19 MB 2025-02-14 08:10:47,962 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -471.86 MB 2025-02-14 08:10:47,962 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18955.74 MB 2025-02-14 08:10:47,972 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 08:10:47,972 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 08:10:47,972 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 08:10:47,972 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:10:47,972 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14977.96 MB 2025-02-14 08:10:47,972 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15667.47 MB 2025-02-14 08:10:47,972 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 689.51 MB 2025-02-14 08:10:47,972 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17899.19 MB 2025-02-14 08:10:47,972 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17899.19 MB 2025-02-14 08:10:47,972 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:10:47,972 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16184.84 MB 2025-02-14 08:10:48,072 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 08:10:48,072 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 08:10:48,072 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 08:10:48,072 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:10:48,072 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15667.47 MB 2025-02-14 08:10:48,072 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16485.79 MB 2025-02-14 08:10:48,072 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 818.32 MB 2025-02-14 08:10:48,072 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17899.19 MB 2025-02-14 08:10:48,072 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19283.31 MB 2025-02-14 08:10:48,072 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1384.12 MB 2025-02-14 08:10:48,072 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18509.41 MB 2025-02-14 08:10:48,074 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 08:10:48,074 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 08:10:48,074 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 08:10:48,074 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:10:48,074 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14977.96 MB 2025-02-14 08:10:48,074 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16485.79 MB 2025-02-14 08:10:48,074 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1507.83 MB 2025-02-14 08:10:48,074 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17899.19 MB 2025-02-14 08:10:48,074 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19283.31 MB 2025-02-14 08:10:48,074 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1384.12 MB 2025-02-14 08:10:48,074 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18509.41 MB 2025-02-14 08:10:48,176 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 08:10:48,176 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 08:10:48,176 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 08:10:48,176 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:10:48,176 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17045.54 MB 2025-02-14 08:10:48,176 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17325.49 MB 2025-02-14 08:10:48,176 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 279.96 MB 2025-02-14 08:10:48,176 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19283.31 MB 2025-02-14 08:10:48,176 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19432.21 MB 2025-02-14 08:10:48,176 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 148.90 MB 2025-02-14 08:10:48,176 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17594.76 MB 2025-02-14 08:10:48,191 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 08:10:48,191 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 08:10:48,191 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:10:48,191 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:10:48,191 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17476.20 MB 2025-02-14 08:10:48,191 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17681.59 MB 2025-02-14 08:10:48,191 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.38 MB 2025-02-14 08:10:48,192 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19432.21 MB 2025-02-14 08:10:48,192 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19436.40 MB 2025-02-14 08:10:48,192 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-14 08:10:48,192 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17686.78 MB 2025-02-14 08:10:48,194 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 08:10:48,194 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 08:10:48,194 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.39 seconds 2025-02-14 08:10:48,194 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:10:48,194 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13498.29 MB 2025-02-14 08:10:48,194 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17882.64 MB 2025-02-14 08:10:48,194 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4384.35 MB 2025-02-14 08:10:48,194 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52378.47 MB 2025-02-14 08:10:48,194 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19436.40 MB 2025-02-14 08:10:48,194 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32942.06 MB 2025-02-14 08:10:48,194 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17882.64 MB 2025-02-14 08:10:48,482 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 08:10:48,482 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 08:10:48,482 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.28 seconds 2025-02-14 08:10:48,482 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:10:48,482 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17882.64 MB 2025-02-14 08:10:48,482 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17303.60 MB 2025-02-14 08:10:48,482 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -579.04 MB 2025-02-14 08:10:48,482 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19436.40 MB 2025-02-14 08:10:48,482 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19704.84 MB 2025-02-14 08:10:48,482 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 268.44 MB 2025-02-14 08:10:48,482 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18988.19 MB 2025-02-14 08:10:48,502 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8161, cut from 8163 2025-02-14 08:10:48,502 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2,'] 2025-02-14 08:10:48,509 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 08:10:48,509 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 08:10:48,509 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:10:48,509 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:10:48,509 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17303.60 MB 2025-02-14 08:10:48,509 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25742.43 MB 2025-02-14 08:10:48,509 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8438.84 MB 2025-02-14 08:10:48,509 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19704.84 MB 2025-02-14 08:10:48,509 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30190.60 MB 2025-02-14 08:10:48,509 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10485.76 MB 2025-02-14 08:10:48,509 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25742.43 MB 2025-02-14 08:10:48,771 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7953] 2025-02-14 08:10:48,774 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:10:48,774 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 08:10:48,775 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:10:48,775 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 08:10:48,783 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 08:10:48,785 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:10:48,785 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 08:10:48,785 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2,'] 2025-02-14 08:10:58,412 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:10:58,412 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 08:10:58,417 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 08:10:58,421 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:10:58,421 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1624, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 08:10:58,422 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:10:58,422 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1624, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 08:11:23,621 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 08:11:23,621 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 08:11:23,621 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.19 seconds 2025-02-14 08:11:23,621 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:11:23,621 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24285.00 MB 2025-02-14 08:11:23,621 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30033.29 MB 2025-02-14 08:11:23,621 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5748.29 MB 2025-02-14 08:11:23,621 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38579.21 MB 2025-02-14 08:11:23,621 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39434.85 MB 2025-02-14 08:11:23,621 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 855.64 MB 2025-02-14 08:11:23,621 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38966.50 MB 2025-02-14 08:11:23,700 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 08:11:23,700 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 08:11:23,700 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 08:11:23,700 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:11:23,700 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30033.29 MB 2025-02-14 08:11:23,700 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24220.52 MB 2025-02-14 08:11:23,700 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5812.77 MB 2025-02-14 08:11:23,700 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39434.85 MB 2025-02-14 08:11:23,700 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49314.53 MB 2025-02-14 08:11:23,700 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9879.68 MB 2025-02-14 08:11:23,700 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41331.40 MB 2025-02-14 08:11:25,630 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 08:11:25,630 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 08:11:25,630 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 08:11:25,630 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:11:25,630 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24220.52 MB 2025-02-14 08:11:25,630 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24751.36 MB 2025-02-14 08:11:25,630 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 08:11:25,630 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49314.53 MB 2025-02-14 08:11:25,630 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30907.83 MB 2025-02-14 08:11:25,630 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18406.70 MB 2025-02-14 08:11:25,630 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28730.70 MB 2025-02-14 08:11:25,645 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 08:11:25,645 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 08:11:25,645 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:11:25,645 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:11:25,645 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24751.36 MB 2025-02-14 08:11:25,645 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26640.90 MB 2025-02-14 08:11:25,645 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 08:11:25,645 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30907.83 MB 2025-02-14 08:11:25,645 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31851.54 MB 2025-02-14 08:11:25,645 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 08:11:25,645 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28058.32 MB 2025-02-14 08:11:25,856 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 08:11:25,856 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 08:11:25,856 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 08:11:25,856 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:11:25,856 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26640.90 MB 2025-02-14 08:11:25,856 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28882.75 MB 2025-02-14 08:11:25,856 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 08:11:25,856 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31851.54 MB 2025-02-14 08:11:25,856 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37513.85 MB 2025-02-14 08:11:25,856 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 08:11:25,856 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34427.03 MB 2025-02-14 08:11:25,857 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 08:11:25,857 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 08:11:25,857 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 08:11:25,857 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:11:25,857 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24751.36 MB 2025-02-14 08:11:25,857 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28882.75 MB 2025-02-14 08:11:25,857 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 08:11:25,857 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30907.83 MB 2025-02-14 08:11:25,857 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37513.85 MB 2025-02-14 08:11:25,857 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 08:11:25,857 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34427.03 MB 2025-02-14 08:11:26,022 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 08:11:26,022 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 08:11:26,022 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 08:11:26,022 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:11:26,022 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30416.29 MB 2025-02-14 08:11:26,022 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31183.30 MB 2025-02-14 08:11:26,022 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 08:11:26,022 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37513.85 MB 2025-02-14 08:11:26,022 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37929.09 MB 2025-02-14 08:11:26,022 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 08:11:26,022 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31891.08 MB 2025-02-14 08:11:26,041 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 08:11:26,041 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 08:11:26,041 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:11:26,041 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:11:26,041 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31596.19 MB 2025-02-14 08:11:26,041 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31822.50 MB 2025-02-14 08:11:26,041 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.32 MB 2025-02-14 08:11:26,041 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37929.09 MB 2025-02-14 08:11:26,041 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37929.09 MB 2025-02-14 08:11:26,041 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:11:26,041 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32044.05 MB 2025-02-14 08:11:26,042 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 08:11:26,042 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 08:11:26,042 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.62 seconds 2025-02-14 08:11:26,042 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:11:26,042 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18626.85 MB 2025-02-14 08:11:26,042 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32022.59 MB 2025-02-14 08:11:26,042 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13395.74 MB 2025-02-14 08:11:26,042 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38579.21 MB 2025-02-14 08:11:26,042 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37929.09 MB 2025-02-14 08:11:26,042 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -650.12 MB 2025-02-14 08:11:26,042 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32044.05 MB 2025-02-14 08:11:26,311 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 08:11:26,311 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 08:11:26,311 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 08:11:26,311 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:11:26,311 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32022.59 MB 2025-02-14 08:11:26,311 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23616.00 MB 2025-02-14 08:11:26,311 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8406.59 MB 2025-02-14 08:11:26,311 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37929.09 MB 2025-02-14 08:11:26,311 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37929.09 MB 2025-02-14 08:11:26,311 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:11:26,311 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34521.97 MB 2025-02-14 08:11:26,329 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8122, cut from 8124 2025-02-14 08:11:26,329 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 08:11:26,335 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 08:11:26,335 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 08:11:26,335 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:11:26,335 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:11:26,335 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23616.00 MB 2025-02-14 08:11:26,335 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32013.41 MB 2025-02-14 08:11:26,335 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8397.40 MB 2025-02-14 08:11:26,335 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37929.09 MB 2025-02-14 08:11:26,335 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42104.52 MB 2025-02-14 08:11:26,335 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4175.43 MB 2025-02-14 08:11:26,335 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32013.41 MB 2025-02-14 08:11:26,497 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7914] 2025-02-14 08:11:26,498 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:11:26,498 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 08:11:26,499 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:11:26,499 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 08:11:26,504 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 08:11:26,505 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:11:26,505 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 08:11:26,505 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 08:11:47,581 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:11:47,581 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 08:11:47,586 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 08:11:47,590 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:11:47,590 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 179, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 08:11:47,591 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:11:47,591 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 179, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 08:11:50,379 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 08:11:50,379 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 08:11:50,379 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.78 seconds 2025-02-14 08:11:50,379 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:11:50,379 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14216.01 MB 2025-02-14 08:11:50,379 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14849.48 MB 2025-02-14 08:11:50,379 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 633.47 MB 2025-02-14 08:11:50,379 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50455.38 MB 2025-02-14 08:11:50,379 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 16909.34 MB 2025-02-14 08:11:50,379 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33546.04 MB 2025-02-14 08:11:50,379 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23688.19 MB 2025-02-14 08:11:50,392 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 08:11:50,392 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 08:11:50,392 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:11:50,392 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:11:50,392 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14849.48 MB 2025-02-14 08:11:50,392 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15037.00 MB 2025-02-14 08:11:50,392 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 187.52 MB 2025-02-14 08:11:50,392 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 16909.34 MB 2025-02-14 08:11:50,392 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18337.50 MB 2025-02-14 08:11:50,392 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1428.16 MB 2025-02-14 08:11:50,392 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17136.50 MB 2025-02-14 08:11:51,191 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 08:11:51,191 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 08:11:51,191 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.80 seconds 2025-02-14 08:11:51,191 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:11:51,191 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15037.00 MB 2025-02-14 08:11:51,191 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15251.99 MB 2025-02-14 08:11:51,191 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 214.99 MB 2025-02-14 08:11:51,191 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18337.50 MB 2025-02-14 08:11:51,191 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17762.88 MB 2025-02-14 08:11:51,191 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -574.62 MB 2025-02-14 08:11:51,191 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19209.52 MB 2025-02-14 08:11:51,199 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 08:11:51,199 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 08:11:51,199 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 08:11:51,199 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:11:51,199 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15251.93 MB 2025-02-14 08:11:51,199 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16017.00 MB 2025-02-14 08:11:51,199 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 765.08 MB 2025-02-14 08:11:51,199 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17762.88 MB 2025-02-14 08:11:51,199 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17762.88 MB 2025-02-14 08:11:51,199 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:11:51,199 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16591.07 MB 2025-02-14 08:11:51,287 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 08:11:51,287 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 08:11:51,287 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 08:11:51,287 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:11:51,287 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16017.00 MB 2025-02-14 08:11:51,287 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16924.99 MB 2025-02-14 08:11:51,287 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 907.99 MB 2025-02-14 08:11:51,287 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17762.88 MB 2025-02-14 08:11:51,287 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20067.65 MB 2025-02-14 08:11:51,287 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2304.77 MB 2025-02-14 08:11:51,287 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19170.39 MB 2025-02-14 08:11:51,287 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 08:11:51,287 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 08:11:51,287 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 08:11:51,288 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:11:51,288 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15251.93 MB 2025-02-14 08:11:51,288 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16924.99 MB 2025-02-14 08:11:51,288 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1673.07 MB 2025-02-14 08:11:51,288 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17762.88 MB 2025-02-14 08:11:51,288 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20067.65 MB 2025-02-14 08:11:51,288 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2304.77 MB 2025-02-14 08:11:51,288 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19170.39 MB 2025-02-14 08:11:51,356 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 08:11:51,357 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 08:11:51,357 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 08:11:51,357 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:11:51,357 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17546.08 MB 2025-02-14 08:11:51,357 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17856.71 MB 2025-02-14 08:11:51,357 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 310.64 MB 2025-02-14 08:11:51,357 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20067.65 MB 2025-02-14 08:11:51,357 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20233.32 MB 2025-02-14 08:11:51,357 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 165.68 MB 2025-02-14 08:11:51,357 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18151.40 MB 2025-02-14 08:11:51,366 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 08:11:51,366 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 08:11:51,366 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:11:51,366 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:11:51,366 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18023.94 MB 2025-02-14 08:11:51,367 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18227.75 MB 2025-02-14 08:11:51,367 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 203.80 MB 2025-02-14 08:11:51,367 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20233.32 MB 2025-02-14 08:11:51,367 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20237.52 MB 2025-02-14 08:11:51,367 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-14 08:11:51,367 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18251.93 MB 2025-02-14 08:11:51,368 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 08:11:51,368 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 08:11:51,368 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.78 seconds 2025-02-14 08:11:51,368 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:11:51,368 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13592.36 MB 2025-02-14 08:11:51,368 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18428.50 MB 2025-02-14 08:11:51,368 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4836.14 MB 2025-02-14 08:11:51,368 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50455.38 MB 2025-02-14 08:11:51,368 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20237.52 MB 2025-02-14 08:11:51,368 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30217.86 MB 2025-02-14 08:11:51,368 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18428.50 MB 2025-02-14 08:11:51,635 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 08:11:51,635 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 08:11:51,635 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 08:11:51,635 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:11:51,635 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18428.50 MB 2025-02-14 08:11:51,635 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17468.28 MB 2025-02-14 08:11:51,635 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -960.22 MB 2025-02-14 08:11:51,635 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20237.52 MB 2025-02-14 08:11:51,635 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20237.52 MB 2025-02-14 08:11:51,635 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:11:51,635 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19130.65 MB 2025-02-14 08:11:51,657 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8149, cut from 8151 2025-02-14 08:11:51,657 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 08:11:51,663 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 08:11:51,663 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 08:11:51,663 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 08:11:51,663 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:11:51,663 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17468.28 MB 2025-02-14 08:11:51,663 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25894.46 MB 2025-02-14 08:11:51,663 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8426.18 MB 2025-02-14 08:11:51,663 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20237.52 MB 2025-02-14 08:11:51,663 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30708.60 MB 2025-02-14 08:11:51,663 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10471.08 MB 2025-02-14 08:11:51,663 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25894.46 MB 2025-02-14 08:11:51,825 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7941] 2025-02-14 08:11:51,826 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:11:51,826 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 08:11:51,827 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:11:51,827 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 08:11:51,832 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 08:11:51,833 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:11:51,833 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 08:11:51,833 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 08:12:19,290 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:12:19,290 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 08:12:19,297 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 08:12:19,303 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:12:19,303 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 469, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 08:12:19,305 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:12:19,305 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 469, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 08:12:26,639 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 08:12:26,639 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 08:12:26,639 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.33 seconds 2025-02-14 08:12:26,639 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:12:26,639 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16236.77 MB 2025-02-14 08:12:26,639 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17896.54 MB 2025-02-14 08:12:26,639 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1659.76 MB 2025-02-14 08:12:26,640 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39084.62 MB 2025-02-14 08:12:26,640 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23190.31 MB 2025-02-14 08:12:26,640 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15894.32 MB 2025-02-14 08:12:26,640 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26840.61 MB 2025-02-14 08:12:26,673 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 08:12:26,673 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 08:12:26,673 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 08:12:26,673 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:12:26,673 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17896.54 MB 2025-02-14 08:12:26,673 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18217.08 MB 2025-02-14 08:12:26,673 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 320.55 MB 2025-02-14 08:12:26,673 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23190.31 MB 2025-02-14 08:12:26,673 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28236.05 MB 2025-02-14 08:12:26,673 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5045.75 MB 2025-02-14 08:12:26,673 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25124.85 MB 2025-02-14 08:12:28,584 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 08:12:28,584 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 08:12:28,584 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 08:12:28,584 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:12:28,584 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18217.08 MB 2025-02-14 08:12:28,584 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18747.93 MB 2025-02-14 08:12:28,584 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 08:12:28,584 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28236.05 MB 2025-02-14 08:12:28,584 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25314.72 MB 2025-02-14 08:12:28,584 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2921.33 MB 2025-02-14 08:12:28,584 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22727.26 MB 2025-02-14 08:12:28,598 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 08:12:28,598 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 08:12:28,598 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:12:28,598 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:12:28,598 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18747.93 MB 2025-02-14 08:12:28,598 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20637.46 MB 2025-02-14 08:12:28,598 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 08:12:28,598 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25314.72 MB 2025-02-14 08:12:28,598 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25314.72 MB 2025-02-14 08:12:28,598 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:12:28,598 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22054.89 MB 2025-02-14 08:12:28,808 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 08:12:28,808 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 08:12:28,808 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 08:12:28,808 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:12:28,808 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20637.46 MB 2025-02-14 08:12:28,808 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22879.32 MB 2025-02-14 08:12:28,808 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 08:12:28,808 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25314.72 MB 2025-02-14 08:12:28,808 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30977.03 MB 2025-02-14 08:12:28,808 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 08:12:28,808 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28423.60 MB 2025-02-14 08:12:28,809 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 08:12:28,809 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 08:12:28,809 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 08:12:28,809 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:12:28,809 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18747.93 MB 2025-02-14 08:12:28,809 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22879.32 MB 2025-02-14 08:12:28,809 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 08:12:28,809 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25314.72 MB 2025-02-14 08:12:28,809 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30977.03 MB 2025-02-14 08:12:28,809 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 08:12:28,809 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28423.60 MB 2025-02-14 08:12:28,967 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 08:12:28,968 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 08:12:28,968 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 08:12:28,968 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:12:28,968 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24412.86 MB 2025-02-14 08:12:28,968 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25179.86 MB 2025-02-14 08:12:28,968 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 08:12:28,968 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30977.03 MB 2025-02-14 08:12:28,968 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31394.37 MB 2025-02-14 08:12:28,968 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 08:12:28,968 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25887.65 MB 2025-02-14 08:12:28,986 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 08:12:28,986 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 08:12:28,986 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:12:28,986 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:12:28,986 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25592.75 MB 2025-02-14 08:12:28,986 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25820.15 MB 2025-02-14 08:12:28,986 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.40 MB 2025-02-14 08:12:28,986 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31394.37 MB 2025-02-14 08:12:28,986 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31394.37 MB 2025-02-14 08:12:28,986 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:12:28,986 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25990.02 MB 2025-02-14 08:12:28,987 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 08:12:28,987 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 08:12:28,987 - resource_logging.py:150 - __exit__ - DEBUG - Time: 9.68 seconds 2025-02-14 08:12:28,987 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:12:28,987 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14602.74 MB 2025-02-14 08:12:28,987 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26021.22 MB 2025-02-14 08:12:28,987 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11418.48 MB 2025-02-14 08:12:28,987 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39084.62 MB 2025-02-14 08:12:28,987 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31394.37 MB 2025-02-14 08:12:28,987 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7690.26 MB 2025-02-14 08:12:28,987 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26021.22 MB 2025-02-14 08:12:29,254 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 08:12:29,254 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 08:12:29,254 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 08:12:29,254 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:12:29,254 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26021.22 MB 2025-02-14 08:12:29,254 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19607.13 MB 2025-02-14 08:12:29,254 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6414.09 MB 2025-02-14 08:12:29,254 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31394.37 MB 2025-02-14 08:12:29,254 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31394.37 MB 2025-02-14 08:12:29,254 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:12:29,254 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28532.89 MB 2025-02-14 08:12:29,272 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 08:12:29,273 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 08:12:29,278 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 08:12:29,279 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 08:12:29,279 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:12:29,279 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:12:29,279 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19607.13 MB 2025-02-14 08:12:29,279 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28046.15 MB 2025-02-14 08:12:29,279 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 08:12:29,279 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31394.37 MB 2025-02-14 08:12:29,279 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39785.07 MB 2025-02-14 08:12:29,279 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 08:12:29,279 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28046.15 MB 2025-02-14 08:12:29,429 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 08:12:29,431 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:12:29,431 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 08:12:29,431 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:12:29,432 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 08:12:29,436 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 08:12:29,437 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:12:29,437 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 08:12:29,437 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 08:12:37,880 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:12:37,880 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 08:12:37,884 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 08:12:37,888 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:12:37,888 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 657, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 08:12:37,889 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:12:37,889 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 657, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 08:12:48,203 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 08:12:48,203 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 08:12:48,203 - resource_logging.py:150 - __exit__ - DEBUG - Time: 10.31 seconds 2025-02-14 08:12:48,203 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:12:48,203 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17546.79 MB 2025-02-14 08:12:48,203 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19871.87 MB 2025-02-14 08:12:48,203 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2325.09 MB 2025-02-14 08:12:48,203 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52370.08 MB 2025-02-14 08:12:48,203 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24607.98 MB 2025-02-14 08:12:48,203 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27762.10 MB 2025-02-14 08:12:48,203 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28830.90 MB 2025-02-14 08:12:48,243 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 08:12:48,243 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 08:12:48,243 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-14 08:12:48,243 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:12:48,243 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19871.87 MB 2025-02-14 08:12:48,243 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19193.39 MB 2025-02-14 08:12:48,244 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -678.48 MB 2025-02-14 08:12:48,244 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24607.98 MB 2025-02-14 08:12:48,244 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31207.72 MB 2025-02-14 08:12:48,244 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6599.74 MB 2025-02-14 08:12:48,244 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27865.59 MB 2025-02-14 08:12:50,176 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 08:12:50,176 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 08:12:50,176 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 08:12:50,176 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:12:50,176 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19193.39 MB 2025-02-14 08:12:50,176 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19724.23 MB 2025-02-14 08:12:50,176 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 08:12:50,176 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31207.72 MB 2025-02-14 08:12:50,176 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23928.50 MB 2025-02-14 08:12:50,176 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7279.21 MB 2025-02-14 08:12:50,176 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23703.56 MB 2025-02-14 08:12:50,190 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 08:12:50,190 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 08:12:50,190 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:12:50,190 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:12:50,190 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19724.23 MB 2025-02-14 08:12:50,190 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21613.76 MB 2025-02-14 08:12:50,190 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 08:12:50,190 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23928.50 MB 2025-02-14 08:12:50,190 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25815.94 MB 2025-02-14 08:12:50,190 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 08:12:50,190 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23031.19 MB 2025-02-14 08:12:50,404 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 08:12:50,404 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 08:12:50,404 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 08:12:50,404 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:12:50,404 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21613.76 MB 2025-02-14 08:12:50,404 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23855.62 MB 2025-02-14 08:12:50,404 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 08:12:50,404 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25815.94 MB 2025-02-14 08:12:50,404 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31950.11 MB 2025-02-14 08:12:50,404 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 08:12:50,404 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29399.90 MB 2025-02-14 08:12:50,405 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 08:12:50,405 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 08:12:50,405 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 08:12:50,405 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:12:50,405 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19724.23 MB 2025-02-14 08:12:50,405 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23855.62 MB 2025-02-14 08:12:50,405 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 08:12:50,405 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23928.50 MB 2025-02-14 08:12:50,405 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31950.11 MB 2025-02-14 08:12:50,405 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-14 08:12:50,405 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29399.90 MB 2025-02-14 08:12:50,579 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 08:12:50,579 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 08:12:50,579 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 08:12:50,579 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:12:50,579 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25389.16 MB 2025-02-14 08:12:50,579 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26156.16 MB 2025-02-14 08:12:50,579 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 08:12:50,579 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31950.11 MB 2025-02-14 08:12:50,579 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32365.35 MB 2025-02-14 08:12:50,579 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 08:12:50,579 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26863.95 MB 2025-02-14 08:12:50,598 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 08:12:50,598 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 08:12:50,598 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:12:50,598 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:12:50,598 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26569.05 MB 2025-02-14 08:12:50,598 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26799.07 MB 2025-02-14 08:12:50,598 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 230.02 MB 2025-02-14 08:12:50,598 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32365.35 MB 2025-02-14 08:12:50,598 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32365.35 MB 2025-02-14 08:12:50,598 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:12:50,598 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26993.96 MB 2025-02-14 08:12:50,599 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 08:12:50,599 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 08:12:50,599 - resource_logging.py:150 - __exit__ - DEBUG - Time: 12.71 seconds 2025-02-14 08:12:50,599 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:12:50,599 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15257.75 MB 2025-02-14 08:12:50,599 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27000.14 MB 2025-02-14 08:12:50,599 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11742.40 MB 2025-02-14 08:12:50,599 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52370.08 MB 2025-02-14 08:12:50,599 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32365.35 MB 2025-02-14 08:12:50,599 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20004.73 MB 2025-02-14 08:12:50,599 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27000.14 MB 2025-02-14 08:12:50,868 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 08:12:50,868 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 08:12:50,868 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 08:12:50,868 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:12:50,868 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27000.14 MB 2025-02-14 08:12:50,868 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20262.14 MB 2025-02-14 08:12:50,868 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6738.01 MB 2025-02-14 08:12:50,868 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32365.35 MB 2025-02-14 08:12:50,868 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32365.35 MB 2025-02-14 08:12:50,868 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:12:50,868 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29511.81 MB 2025-02-14 08:12:50,886 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 08:12:50,887 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 08:12:50,893 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 08:12:50,893 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 08:12:50,893 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:12:50,893 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:12:50,893 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20262.14 MB 2025-02-14 08:12:50,893 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28701.16 MB 2025-02-14 08:12:50,893 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 08:12:50,893 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32365.35 MB 2025-02-14 08:12:50,893 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40756.05 MB 2025-02-14 08:12:50,893 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 08:12:50,893 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28701.16 MB 2025-02-14 08:12:51,062 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 08:12:51,064 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:12:51,064 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 08:12:51,065 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:12:51,065 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 08:12:51,070 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 08:12:51,071 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:12:51,071 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 08:12:51,071 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 08:13:23,560 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:13:23,560 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 08:13:23,564 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 08:13:23,569 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:13:23,569 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 153, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 08:13:23,570 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:13:23,570 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 153, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 08:13:25,990 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 08:13:25,990 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 08:13:25,990 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.42 seconds 2025-02-14 08:13:25,990 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:13:25,990 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14034.84 MB 2025-02-14 08:13:25,990 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14576.29 MB 2025-02-14 08:13:25,990 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 541.46 MB 2025-02-14 08:13:25,990 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53341.06 MB 2025-02-14 08:13:25,990 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18796.77 MB 2025-02-14 08:13:25,990 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34544.29 MB 2025-02-14 08:13:25,990 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23507.01 MB 2025-02-14 08:13:26,002 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 08:13:26,002 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 08:13:26,002 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:13:26,002 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:13:26,002 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14576.29 MB 2025-02-14 08:13:26,002 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14712.21 MB 2025-02-14 08:13:26,002 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 135.92 MB 2025-02-14 08:13:26,002 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18796.77 MB 2025-02-14 08:13:26,002 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18796.77 MB 2025-02-14 08:13:26,002 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:13:26,002 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16504.43 MB 2025-02-14 08:13:26,664 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 08:13:26,664 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 08:13:26,664 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.66 seconds 2025-02-14 08:13:26,664 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:13:26,664 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14712.21 MB 2025-02-14 08:13:26,664 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14891.37 MB 2025-02-14 08:13:26,664 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 179.16 MB 2025-02-14 08:13:26,664 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18796.77 MB 2025-02-14 08:13:26,664 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18796.77 MB 2025-02-14 08:13:26,664 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:13:26,664 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18883.69 MB 2025-02-14 08:13:26,671 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 08:13:26,671 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 08:13:26,671 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 08:13:26,671 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:13:26,671 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14891.31 MB 2025-02-14 08:13:26,671 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15528.87 MB 2025-02-14 08:13:26,671 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 637.56 MB 2025-02-14 08:13:26,671 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18796.77 MB 2025-02-14 08:13:26,671 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18796.77 MB 2025-02-14 08:13:26,671 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:13:26,671 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16007.26 MB 2025-02-14 08:13:26,744 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 08:13:26,744 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 08:13:26,744 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 08:13:26,744 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:13:26,744 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15528.87 MB 2025-02-14 08:13:26,744 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16285.54 MB 2025-02-14 08:13:26,744 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 756.67 MB 2025-02-14 08:13:26,744 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18796.77 MB 2025-02-14 08:13:26,744 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19115.54 MB 2025-02-14 08:13:26,744 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 318.77 MB 2025-02-14 08:13:26,744 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18156.95 MB 2025-02-14 08:13:26,745 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 08:13:26,745 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 08:13:26,745 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 08:13:26,745 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:13:26,745 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14891.31 MB 2025-02-14 08:13:26,745 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16285.54 MB 2025-02-14 08:13:26,745 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1394.23 MB 2025-02-14 08:13:26,745 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18796.77 MB 2025-02-14 08:13:26,745 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19115.54 MB 2025-02-14 08:13:26,745 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 318.77 MB 2025-02-14 08:13:26,745 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18156.95 MB 2025-02-14 08:13:26,802 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 08:13:26,802 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 08:13:26,802 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-14 08:13:26,802 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:13:26,802 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16803.11 MB 2025-02-14 08:13:26,802 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17061.97 MB 2025-02-14 08:13:26,802 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 258.86 MB 2025-02-14 08:13:26,802 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19115.54 MB 2025-02-14 08:13:26,802 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19251.86 MB 2025-02-14 08:13:26,802 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 136.31 MB 2025-02-14 08:13:26,802 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17312.81 MB 2025-02-14 08:13:26,810 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 08:13:26,811 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 08:13:26,811 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:13:26,811 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:13:26,811 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17201.33 MB 2025-02-14 08:13:26,811 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17423.55 MB 2025-02-14 08:13:26,811 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 222.21 MB 2025-02-14 08:13:26,811 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19251.86 MB 2025-02-14 08:13:26,811 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19251.86 MB 2025-02-14 08:13:26,811 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:13:26,811 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17423.55 MB 2025-02-14 08:13:26,812 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 08:13:26,812 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 08:13:26,812 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.24 seconds 2025-02-14 08:13:26,812 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:13:26,812 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13501.77 MB 2025-02-14 08:13:26,812 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14240.84 MB 2025-02-14 08:13:26,813 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 739.07 MB 2025-02-14 08:13:26,813 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53341.06 MB 2025-02-14 08:13:26,813 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19251.86 MB 2025-02-14 08:13:26,813 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34089.21 MB 2025-02-14 08:13:26,813 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17623.27 MB 2025-02-14 08:13:27,078 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 08:13:27,078 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 08:13:27,078 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 08:13:27,078 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:13:27,078 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14240.84 MB 2025-02-14 08:13:27,078 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17234.59 MB 2025-02-14 08:13:27,078 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2993.76 MB 2025-02-14 08:13:27,078 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19251.86 MB 2025-02-14 08:13:27,078 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19251.86 MB 2025-02-14 08:13:27,078 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:13:27,078 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17533.93 MB 2025-02-14 08:13:27,096 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8107, cut from 8109 2025-02-14 08:13:27,096 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2,'] 2025-02-14 08:13:27,102 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 08:13:27,102 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 08:13:27,102 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:13:27,103 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:13:27,103 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17234.59 MB 2025-02-14 08:13:27,103 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25617.39 MB 2025-02-14 08:13:27,103 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8382.79 MB 2025-02-14 08:13:27,103 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19251.86 MB 2025-02-14 08:13:27,103 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29670.51 MB 2025-02-14 08:13:27,103 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10418.65 MB 2025-02-14 08:13:27,103 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25617.39 MB 2025-02-14 08:13:27,254 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7899] 2025-02-14 08:13:27,255 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:13:27,255 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 08:13:27,256 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:13:27,256 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 08:13:27,261 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 08:13:27,262 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:13:27,262 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 08:13:27,262 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2,'] 2025-02-14 08:13:35,282 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:13:35,282 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 08:13:35,287 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 08:13:35,291 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:13:35,291 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 716, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 08:13:35,292 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:13:35,292 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 716, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 08:13:46,429 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 08:13:46,429 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 08:13:46,429 - resource_logging.py:150 - __exit__ - DEBUG - Time: 11.13 seconds 2025-02-14 08:13:46,429 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:13:46,429 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23342.14 MB 2025-02-14 08:13:46,429 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25876.02 MB 2025-02-14 08:13:46,429 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2533.88 MB 2025-02-14 08:13:46,429 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38004.59 MB 2025-02-14 08:13:46,429 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31281.12 MB 2025-02-14 08:13:46,429 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6723.47 MB 2025-02-14 08:13:46,429 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34852.75 MB 2025-02-14 08:13:46,476 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 08:13:46,476 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 08:13:46,476 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-14 08:13:46,476 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:13:46,476 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25876.02 MB 2025-02-14 08:13:46,476 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24885.39 MB 2025-02-14 08:13:46,476 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -990.63 MB 2025-02-14 08:13:46,476 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31281.12 MB 2025-02-14 08:13:46,476 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39655.05 MB 2025-02-14 08:13:46,476 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8373.93 MB 2025-02-14 08:13:46,476 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34978.26 MB 2025-02-14 08:13:48,398 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 08:13:48,399 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 08:13:48,399 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 08:13:48,399 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:13:48,399 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24885.39 MB 2025-02-14 08:13:48,399 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25416.23 MB 2025-02-14 08:13:48,399 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 08:13:48,399 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39655.05 MB 2025-02-14 08:13:48,399 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30870.08 MB 2025-02-14 08:13:48,399 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8784.97 MB 2025-02-14 08:13:48,399 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29395.57 MB 2025-02-14 08:13:48,414 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 08:13:48,414 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 08:13:48,414 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:13:48,414 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:13:48,414 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25416.23 MB 2025-02-14 08:13:48,414 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27305.77 MB 2025-02-14 08:13:48,414 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 08:13:48,414 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30870.08 MB 2025-02-14 08:13:48,414 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33701.23 MB 2025-02-14 08:13:48,414 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-14 08:13:48,414 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28723.20 MB 2025-02-14 08:13:48,626 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 08:13:48,626 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 08:13:48,626 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 08:13:48,626 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:13:48,626 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27305.77 MB 2025-02-14 08:13:48,626 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24163.39 MB 2025-02-14 08:13:48,626 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3142.38 MB 2025-02-14 08:13:48,626 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33701.23 MB 2025-02-14 08:13:48,626 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33701.23 MB 2025-02-14 08:13:48,626 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:13:48,626 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29707.67 MB 2025-02-14 08:13:48,627 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 08:13:48,627 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 08:13:48,627 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 08:13:48,627 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:13:48,627 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25416.23 MB 2025-02-14 08:13:48,627 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24163.39 MB 2025-02-14 08:13:48,627 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1252.84 MB 2025-02-14 08:13:48,627 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30870.08 MB 2025-02-14 08:13:48,627 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33701.23 MB 2025-02-14 08:13:48,627 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-14 08:13:48,627 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29707.67 MB 2025-02-14 08:13:48,786 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 08:13:48,786 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 08:13:48,786 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 08:13:48,786 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:13:48,786 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25696.93 MB 2025-02-14 08:13:48,786 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26463.94 MB 2025-02-14 08:13:48,786 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 08:13:48,786 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33701.23 MB 2025-02-14 08:13:48,786 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34114.37 MB 2025-02-14 08:13:48,786 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 08:13:48,786 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27171.72 MB 2025-02-14 08:13:48,806 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 08:13:48,806 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 08:13:48,806 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:13:48,806 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:13:48,806 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26876.83 MB 2025-02-14 08:13:48,806 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27106.40 MB 2025-02-14 08:13:48,806 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.58 MB 2025-02-14 08:13:48,806 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34114.37 MB 2025-02-14 08:13:48,806 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34114.37 MB 2025-02-14 08:13:48,806 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:13:48,806 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27299.70 MB 2025-02-14 08:13:48,807 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 08:13:48,807 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 08:13:48,807 - resource_logging.py:150 - __exit__ - DEBUG - Time: 13.51 seconds 2025-02-14 08:13:48,807 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:13:48,807 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20847.54 MB 2025-02-14 08:13:48,807 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27307.47 MB 2025-02-14 08:13:48,807 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6459.93 MB 2025-02-14 08:13:48,807 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38004.59 MB 2025-02-14 08:13:48,807 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34114.37 MB 2025-02-14 08:13:48,807 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3890.22 MB 2025-02-14 08:13:48,807 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27307.47 MB 2025-02-14 08:13:49,075 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 08:13:49,075 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 08:13:49,075 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 08:13:49,075 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:13:49,075 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27307.47 MB 2025-02-14 08:13:49,075 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20467.70 MB 2025-02-14 08:13:49,075 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6839.78 MB 2025-02-14 08:13:49,075 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34114.37 MB 2025-02-14 08:13:49,075 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34114.37 MB 2025-02-14 08:13:49,075 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:13:49,075 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29819.14 MB 2025-02-14 08:13:49,093 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 08:13:49,094 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 08:13:49,100 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 08:13:49,100 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 08:13:49,100 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:13:49,100 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:13:49,100 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20467.70 MB 2025-02-14 08:13:49,100 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28906.72 MB 2025-02-14 08:13:49,100 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 08:13:49,100 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34114.37 MB 2025-02-14 08:13:49,100 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42505.08 MB 2025-02-14 08:13:49,100 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 08:13:49,100 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28906.72 MB 2025-02-14 08:13:49,252 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 08:13:49,253 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:13:49,253 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 08:13:49,254 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:13:49,254 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 08:13:49,259 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 08:13:49,260 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:13:49,260 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 08:13:49,260 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 08:14:35,262 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:14:35,263 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 08:14:35,268 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 08:14:35,272 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:14:35,272 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 159, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 08:14:35,273 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:14:35,273 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 159, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 08:14:37,742 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 08:14:37,742 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 08:14:37,742 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.46 seconds 2025-02-14 08:14:37,742 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:14:37,742 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14076.64 MB 2025-02-14 08:14:37,742 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14639.34 MB 2025-02-14 08:14:37,742 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 562.69 MB 2025-02-14 08:14:37,742 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55090.09 MB 2025-02-14 08:14:37,742 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18324.91 MB 2025-02-14 08:14:37,742 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -36765.17 MB 2025-02-14 08:14:37,742 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23548.82 MB 2025-02-14 08:14:37,754 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 08:14:37,754 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 08:14:37,754 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:14:37,754 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:14:37,754 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14639.34 MB 2025-02-14 08:14:37,754 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14701.27 MB 2025-02-14 08:14:37,754 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 61.93 MB 2025-02-14 08:14:37,754 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18324.91 MB 2025-02-14 08:14:37,754 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18324.91 MB 2025-02-14 08:14:37,754 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:14:37,754 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16504.43 MB 2025-02-14 08:14:38,380 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 08:14:38,380 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 08:14:38,380 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.62 seconds 2025-02-14 08:14:38,380 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:14:38,380 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14701.27 MB 2025-02-14 08:14:38,380 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14872.46 MB 2025-02-14 08:14:38,380 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 171.20 MB 2025-02-14 08:14:38,380 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18324.91 MB 2025-02-14 08:14:38,380 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18324.91 MB 2025-02-14 08:14:38,380 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:14:38,380 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18872.74 MB 2025-02-14 08:14:38,387 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 08:14:38,387 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 08:14:38,387 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 08:14:38,387 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:14:38,387 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14872.40 MB 2025-02-14 08:14:38,387 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15481.63 MB 2025-02-14 08:14:38,387 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 609.23 MB 2025-02-14 08:14:38,387 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18324.91 MB 2025-02-14 08:14:38,387 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18324.91 MB 2025-02-14 08:14:38,387 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:14:38,387 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15938.75 MB 2025-02-14 08:14:38,457 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 08:14:38,457 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 08:14:38,457 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 08:14:38,457 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:14:38,457 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15481.63 MB 2025-02-14 08:14:38,457 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16204.67 MB 2025-02-14 08:14:38,457 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 723.04 MB 2025-02-14 08:14:38,457 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18324.91 MB 2025-02-14 08:14:38,457 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19243.47 MB 2025-02-14 08:14:38,457 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 918.55 MB 2025-02-14 08:14:38,457 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17992.66 MB 2025-02-14 08:14:38,458 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 08:14:38,458 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 08:14:38,458 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 08:14:38,458 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:14:38,458 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14872.40 MB 2025-02-14 08:14:38,458 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16204.67 MB 2025-02-14 08:14:38,458 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1332.27 MB 2025-02-14 08:14:38,458 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18324.91 MB 2025-02-14 08:14:38,458 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19243.47 MB 2025-02-14 08:14:38,458 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 918.55 MB 2025-02-14 08:14:38,458 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17992.66 MB 2025-02-14 08:14:38,512 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 08:14:38,512 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 08:14:38,512 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-14 08:14:38,512 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:14:38,512 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16699.24 MB 2025-02-14 08:14:38,512 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16946.59 MB 2025-02-14 08:14:38,512 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 247.36 MB 2025-02-14 08:14:38,512 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19243.47 MB 2025-02-14 08:14:38,512 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19373.49 MB 2025-02-14 08:14:38,512 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 130.02 MB 2025-02-14 08:14:38,512 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17187.44 MB 2025-02-14 08:14:38,520 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 08:14:38,520 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 08:14:38,520 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:14:38,520 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:14:38,520 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17079.76 MB 2025-02-14 08:14:38,520 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17283.93 MB 2025-02-14 08:14:38,520 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 204.17 MB 2025-02-14 08:14:38,520 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19373.49 MB 2025-02-14 08:14:38,520 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19373.49 MB 2025-02-14 08:14:38,520 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:14:38,520 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17283.93 MB 2025-02-14 08:14:38,521 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 08:14:38,521 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 08:14:38,521 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.25 seconds 2025-02-14 08:14:38,521 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:14:38,521 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13522.68 MB 2025-02-14 08:14:38,521 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17463.46 MB 2025-02-14 08:14:38,521 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3940.78 MB 2025-02-14 08:14:38,521 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55090.09 MB 2025-02-14 08:14:38,521 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19373.49 MB 2025-02-14 08:14:38,521 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35716.60 MB 2025-02-14 08:14:38,521 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17463.46 MB 2025-02-14 08:14:38,761 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 08:14:38,761 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 08:14:38,761 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.24 seconds 2025-02-14 08:14:38,761 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:14:38,761 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17463.46 MB 2025-02-14 08:14:38,761 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16914.43 MB 2025-02-14 08:14:38,761 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -549.03 MB 2025-02-14 08:14:38,761 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19373.49 MB 2025-02-14 08:14:38,761 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19373.49 MB 2025-02-14 08:14:38,761 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:14:38,761 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18719.29 MB 2025-02-14 08:14:38,777 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 7286, cut from 7288 2025-02-14 08:14:38,777 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 08:14:38,783 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 08:14:38,783 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 08:14:38,783 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:14:38,783 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:14:38,784 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16914.43 MB 2025-02-14 08:14:38,784 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24450.45 MB 2025-02-14 08:14:38,784 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7536.03 MB 2025-02-14 08:14:38,784 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19373.49 MB 2025-02-14 08:14:38,784 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28737.27 MB 2025-02-14 08:14:38,784 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9363.78 MB 2025-02-14 08:14:38,784 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24450.45 MB 2025-02-14 08:14:38,924 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7078] 2025-02-14 08:14:38,926 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:14:38,926 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 08:14:38,927 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:14:38,927 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 08:14:38,931 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 08:14:38,932 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:14:38,932 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 08:14:38,932 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 08:15:59,352 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:15:59,352 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 08:15:59,360 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 08:15:59,367 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:15:59,367 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1157, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 08:15:59,369 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:15:59,369 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1157, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 08:16:17,212 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 08:16:17,212 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 08:16:17,212 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.83 seconds 2025-02-14 08:16:17,212 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:16:17,212 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21030.87 MB 2025-02-14 08:16:17,212 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25125.43 MB 2025-02-14 08:16:17,212 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4094.56 MB 2025-02-14 08:16:17,212 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36228.30 MB 2025-02-14 08:16:17,212 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35982.93 MB 2025-02-14 08:16:17,212 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -245.37 MB 2025-02-14 08:16:17,212 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34126.12 MB 2025-02-14 08:16:17,279 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 08:16:17,279 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 08:16:17,280 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 08:16:17,280 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:16:17,280 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25125.43 MB 2025-02-14 08:16:17,280 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21792.73 MB 2025-02-14 08:16:17,280 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3332.69 MB 2025-02-14 08:16:17,280 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35982.93 MB 2025-02-14 08:16:17,280 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44111.50 MB 2025-02-14 08:16:17,280 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8128.56 MB 2025-02-14 08:16:17,280 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37494.52 MB 2025-02-14 08:16:19,193 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 08:16:19,193 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 08:16:19,193 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 08:16:19,193 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:16:19,193 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21792.73 MB 2025-02-14 08:16:19,193 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22323.57 MB 2025-02-14 08:16:19,193 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 08:16:19,193 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44111.50 MB 2025-02-14 08:16:19,193 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28141.68 MB 2025-02-14 08:16:19,193 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15969.81 MB 2025-02-14 08:16:19,194 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26302.91 MB 2025-02-14 08:16:19,207 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 08:16:19,207 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 08:16:19,207 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:16:19,207 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:16:19,207 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22323.57 MB 2025-02-14 08:16:19,207 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24213.11 MB 2025-02-14 08:16:19,207 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 08:16:19,207 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28141.68 MB 2025-02-14 08:16:19,207 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29087.50 MB 2025-02-14 08:16:19,207 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 945.82 MB 2025-02-14 08:16:19,207 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25630.54 MB 2025-02-14 08:16:19,419 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 08:16:19,419 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 08:16:19,419 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 08:16:19,419 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:16:19,419 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24213.11 MB 2025-02-14 08:16:19,419 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26454.96 MB 2025-02-14 08:16:19,419 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 08:16:19,419 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29087.50 MB 2025-02-14 08:16:19,419 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34749.81 MB 2025-02-14 08:16:19,419 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 08:16:19,419 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31999.25 MB 2025-02-14 08:16:19,419 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 08:16:19,419 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 08:16:19,419 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 08:16:19,419 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:16:19,419 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22323.57 MB 2025-02-14 08:16:19,419 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26454.96 MB 2025-02-14 08:16:19,419 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 08:16:19,419 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28141.68 MB 2025-02-14 08:16:19,419 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34749.81 MB 2025-02-14 08:16:19,419 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6608.13 MB 2025-02-14 08:16:19,419 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31999.25 MB 2025-02-14 08:16:19,588 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 08:16:19,588 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 08:16:19,588 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 08:16:19,588 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:16:19,588 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27988.51 MB 2025-02-14 08:16:19,588 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28755.51 MB 2025-02-14 08:16:19,588 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 08:16:19,589 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34749.81 MB 2025-02-14 08:16:19,589 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35167.14 MB 2025-02-14 08:16:19,589 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 08:16:19,589 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29463.30 MB 2025-02-14 08:16:19,608 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 08:16:19,608 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 08:16:19,608 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:16:19,608 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:16:19,608 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29168.40 MB 2025-02-14 08:16:19,608 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29397.35 MB 2025-02-14 08:16:19,608 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.95 MB 2025-02-14 08:16:19,608 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35167.14 MB 2025-02-14 08:16:19,608 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35167.14 MB 2025-02-14 08:16:19,608 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:16:19,608 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29644.16 MB 2025-02-14 08:16:19,609 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 08:16:19,609 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 08:16:19,609 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.24 seconds 2025-02-14 08:16:19,609 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:16:19,609 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16999.79 MB 2025-02-14 08:16:19,609 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29597.86 MB 2025-02-14 08:16:19,609 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12598.07 MB 2025-02-14 08:16:19,609 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36228.30 MB 2025-02-14 08:16:19,609 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35167.14 MB 2025-02-14 08:16:19,609 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1061.16 MB 2025-02-14 08:16:19,609 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29644.16 MB 2025-02-14 08:16:19,877 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 08:16:19,877 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 08:16:19,877 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 08:16:19,877 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:16:19,877 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29597.86 MB 2025-02-14 08:16:19,877 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21995.42 MB 2025-02-14 08:16:19,877 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7602.44 MB 2025-02-14 08:16:19,877 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35167.14 MB 2025-02-14 08:16:19,877 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35167.14 MB 2025-02-14 08:16:19,877 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:16:19,877 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32102.46 MB 2025-02-14 08:16:19,895 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8139, cut from 8141 2025-02-14 08:16:19,895 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 08:16:19,901 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 08:16:19,901 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 08:16:19,901 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:16:19,901 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:16:19,901 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21995.42 MB 2025-02-14 08:16:19,901 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30410.36 MB 2025-02-14 08:16:19,901 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8414.95 MB 2025-02-14 08:16:19,901 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35167.14 MB 2025-02-14 08:16:19,901 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43534.78 MB 2025-02-14 08:16:19,901 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8367.64 MB 2025-02-14 08:16:19,901 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30410.36 MB 2025-02-14 08:16:20,064 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7931] 2025-02-14 08:16:20,065 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:16:20,065 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 08:16:20,066 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:16:20,066 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 08:16:20,071 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 08:16:20,072 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:16:20,072 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 08:16:20,072 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 08:17:33,650 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:17:33,650 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 08:17:33,654 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 08:17:33,658 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:17:33,658 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1710, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 08:17:33,659 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:17:33,659 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1710, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 08:18:00,000 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 08:18:00,001 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 08:18:00,001 - resource_logging.py:150 - __exit__ - DEBUG - Time: 26.33 seconds 2025-02-14 08:18:00,001 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:18:00,001 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24884.26 MB 2025-02-14 08:18:00,001 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30936.64 MB 2025-02-14 08:18:00,001 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6052.38 MB 2025-02-14 08:18:00,001 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51902.41 MB 2025-02-14 08:18:00,001 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38818.28 MB 2025-02-14 08:18:00,001 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13084.13 MB 2025-02-14 08:18:00,001 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39791.45 MB 2025-02-14 08:18:00,106 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 08:18:00,106 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 08:18:00,106 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 08:18:00,106 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:18:00,106 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30936.64 MB 2025-02-14 08:18:00,106 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24667.61 MB 2025-02-14 08:18:00,106 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6269.03 MB 2025-02-14 08:18:00,106 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38818.28 MB 2025-02-14 08:18:00,106 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57749.27 MB 2025-02-14 08:18:00,106 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 18930.99 MB 2025-02-14 08:18:00,106 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48753.31 MB 2025-02-14 08:18:02,040 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 08:18:02,040 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 08:18:02,040 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 08:18:02,040 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:18:02,040 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24667.61 MB 2025-02-14 08:18:02,040 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25198.45 MB 2025-02-14 08:18:02,040 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 08:18:02,040 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57749.27 MB 2025-02-14 08:18:02,040 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34181.48 MB 2025-02-14 08:18:02,040 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23567.79 MB 2025-02-14 08:18:02,040 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29177.78 MB 2025-02-14 08:18:02,053 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 08:18:02,053 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 08:18:02,053 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:18:02,053 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:18:02,053 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25198.45 MB 2025-02-14 08:18:02,054 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27087.98 MB 2025-02-14 08:18:02,054 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 08:18:02,054 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34181.48 MB 2025-02-14 08:18:02,054 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34181.48 MB 2025-02-14 08:18:02,054 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:18:02,054 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28505.41 MB 2025-02-14 08:18:02,264 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 08:18:02,264 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 08:18:02,265 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 08:18:02,265 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:18:02,265 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27087.98 MB 2025-02-14 08:18:02,265 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29329.84 MB 2025-02-14 08:18:02,265 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 08:18:02,265 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34181.48 MB 2025-02-14 08:18:02,265 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37956.35 MB 2025-02-14 08:18:02,265 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 08:18:02,265 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34874.12 MB 2025-02-14 08:18:02,265 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 08:18:02,265 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 08:18:02,265 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 08:18:02,265 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:18:02,265 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25198.45 MB 2025-02-14 08:18:02,265 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29329.84 MB 2025-02-14 08:18:02,265 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 08:18:02,265 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34181.48 MB 2025-02-14 08:18:02,265 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37956.35 MB 2025-02-14 08:18:02,265 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 08:18:02,265 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34874.12 MB 2025-02-14 08:18:02,435 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 08:18:02,435 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 08:18:02,435 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 08:18:02,435 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:18:02,435 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30863.38 MB 2025-02-14 08:18:02,435 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31630.38 MB 2025-02-14 08:18:02,435 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 08:18:02,435 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37956.35 MB 2025-02-14 08:18:02,435 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38373.69 MB 2025-02-14 08:18:02,435 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 08:18:02,435 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32338.17 MB 2025-02-14 08:18:02,454 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 08:18:02,455 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 08:18:02,455 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:18:02,455 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:18:02,455 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32043.27 MB 2025-02-14 08:18:02,455 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32271.96 MB 2025-02-14 08:18:02,455 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.69 MB 2025-02-14 08:18:02,455 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38373.69 MB 2025-02-14 08:18:02,455 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38373.69 MB 2025-02-14 08:18:02,455 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:18:02,455 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32498.52 MB 2025-02-14 08:18:02,456 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 08:18:02,456 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 08:18:02,456 - resource_logging.py:150 - __exit__ - DEBUG - Time: 28.79 seconds 2025-02-14 08:18:02,456 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:18:02,456 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18926.48 MB 2025-02-14 08:18:02,456 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32472.54 MB 2025-02-14 08:18:02,456 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13546.06 MB 2025-02-14 08:18:02,456 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51902.41 MB 2025-02-14 08:18:02,456 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38373.69 MB 2025-02-14 08:18:02,456 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13528.73 MB 2025-02-14 08:18:02,456 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32498.52 MB 2025-02-14 08:18:02,724 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 08:18:02,724 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 08:18:02,724 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 08:18:02,724 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:18:02,724 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32472.54 MB 2025-02-14 08:18:02,724 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23923.25 MB 2025-02-14 08:18:02,724 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8549.29 MB 2025-02-14 08:18:02,724 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38373.69 MB 2025-02-14 08:18:02,724 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38373.69 MB 2025-02-14 08:18:02,724 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:18:02,724 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34978.07 MB 2025-02-14 08:18:02,742 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8142, cut from 8144 2025-02-14 08:18:02,742 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 08:18:02,748 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 08:18:02,748 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 08:18:02,748 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:18:02,748 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:18:02,748 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23923.25 MB 2025-02-14 08:18:02,748 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32341.41 MB 2025-02-14 08:18:02,748 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8418.15 MB 2025-02-14 08:18:02,748 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38373.69 MB 2025-02-14 08:18:02,748 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46743.42 MB 2025-02-14 08:18:02,748 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8369.73 MB 2025-02-14 08:18:02,748 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32341.41 MB 2025-02-14 08:18:02,912 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7934] 2025-02-14 08:18:02,913 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:18:02,913 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 08:18:02,914 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:18:02,914 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 08:18:02,920 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 08:18:02,921 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:18:02,921 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 08:18:02,921 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 08:18:12,899 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:18:12,900 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 08:18:12,904 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 08:18:12,908 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:18:12,908 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1880, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 08:18:12,909 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:18:12,909 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1880, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 08:18:42,204 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 08:18:42,205 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 08:18:42,205 - resource_logging.py:150 - __exit__ - DEBUG - Time: 29.29 seconds 2025-02-14 08:18:42,205 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:18:42,205 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26068.85 MB 2025-02-14 08:18:42,205 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32723.11 MB 2025-02-14 08:18:42,205 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6654.26 MB 2025-02-14 08:18:42,205 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59296.97 MB 2025-02-14 08:18:42,205 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39420.17 MB 2025-02-14 08:18:42,205 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19876.81 MB 2025-02-14 08:18:42,205 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41656.32 MB 2025-02-14 08:18:42,344 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 08:18:42,345 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 08:18:42,345 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-14 08:18:42,345 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:18:42,345 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32723.11 MB 2025-02-14 08:18:42,345 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25551.38 MB 2025-02-14 08:18:42,345 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7171.73 MB 2025-02-14 08:18:42,345 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39420.17 MB 2025-02-14 08:18:42,345 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 62212.01 MB 2025-02-14 08:18:42,345 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 22791.85 MB 2025-02-14 08:18:42,345 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52492.78 MB 2025-02-14 08:18:44,279 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 08:18:44,279 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 08:18:44,280 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 08:18:44,280 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:18:44,280 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25551.38 MB 2025-02-14 08:18:44,280 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26082.23 MB 2025-02-14 08:18:44,280 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 08:18:44,280 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62212.01 MB 2025-02-14 08:18:44,280 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34181.48 MB 2025-02-14 08:18:44,280 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28030.53 MB 2025-02-14 08:18:44,280 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30061.56 MB 2025-02-14 08:18:44,293 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 08:18:44,293 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 08:18:44,293 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:18:44,293 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:18:44,293 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26082.23 MB 2025-02-14 08:18:44,293 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27971.76 MB 2025-02-14 08:18:44,293 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 08:18:44,293 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34181.48 MB 2025-02-14 08:18:44,293 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34181.48 MB 2025-02-14 08:18:44,293 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:18:44,293 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29389.19 MB 2025-02-14 08:18:44,503 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 08:18:44,503 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 08:18:44,503 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 08:18:44,503 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:18:44,503 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27971.76 MB 2025-02-14 08:18:44,503 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30213.62 MB 2025-02-14 08:18:44,503 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 08:18:44,503 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34181.48 MB 2025-02-14 08:18:44,503 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38428.21 MB 2025-02-14 08:18:44,503 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4246.73 MB 2025-02-14 08:18:44,503 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35757.90 MB 2025-02-14 08:18:44,504 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 08:18:44,504 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 08:18:44,504 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 08:18:44,504 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:18:44,504 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26082.23 MB 2025-02-14 08:18:44,504 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30213.62 MB 2025-02-14 08:18:44,504 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 08:18:44,504 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34181.48 MB 2025-02-14 08:18:44,504 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38428.21 MB 2025-02-14 08:18:44,504 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4246.73 MB 2025-02-14 08:18:44,504 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35757.90 MB 2025-02-14 08:18:44,671 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 08:18:44,671 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 08:18:44,671 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 08:18:44,671 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:18:44,671 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31747.16 MB 2025-02-14 08:18:44,671 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32514.16 MB 2025-02-14 08:18:44,671 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 08:18:44,671 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38428.21 MB 2025-02-14 08:18:44,671 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38845.55 MB 2025-02-14 08:18:44,671 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 08:18:44,671 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33221.95 MB 2025-02-14 08:18:44,690 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 08:18:44,690 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 08:18:44,690 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:18:44,690 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:18:44,690 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32927.05 MB 2025-02-14 08:18:44,690 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33155.05 MB 2025-02-14 08:18:44,690 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.00 MB 2025-02-14 08:18:44,690 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38845.55 MB 2025-02-14 08:18:44,690 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38845.55 MB 2025-02-14 08:18:44,690 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:18:44,690 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33394.28 MB 2025-02-14 08:18:44,691 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 08:18:44,691 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 08:18:44,691 - resource_logging.py:150 - __exit__ - DEBUG - Time: 31.78 seconds 2025-02-14 08:18:44,691 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:18:44,691 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19518.78 MB 2025-02-14 08:18:44,691 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33354.97 MB 2025-02-14 08:18:44,691 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13836.19 MB 2025-02-14 08:18:44,691 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59296.97 MB 2025-02-14 08:18:44,691 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38845.55 MB 2025-02-14 08:18:44,691 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20451.43 MB 2025-02-14 08:18:44,691 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33394.28 MB 2025-02-14 08:18:44,962 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 08:18:44,962 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 08:18:44,962 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 08:18:44,962 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:18:44,962 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33354.97 MB 2025-02-14 08:18:44,962 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24505.26 MB 2025-02-14 08:18:44,962 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8849.71 MB 2025-02-14 08:18:44,962 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38845.55 MB 2025-02-14 08:18:44,962 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38845.55 MB 2025-02-14 08:18:44,962 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:18:44,962 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35852.20 MB 2025-02-14 08:18:44,980 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8115, cut from 8117 2025-02-14 08:18:44,980 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 08:18:44,986 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 08:18:44,986 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 08:18:44,986 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:18:44,986 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:18:44,986 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24505.26 MB 2025-02-14 08:18:44,986 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32895.44 MB 2025-02-14 08:18:44,986 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8390.18 MB 2025-02-14 08:18:44,986 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38845.55 MB 2025-02-14 08:18:44,986 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43016.78 MB 2025-02-14 08:18:44,987 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4171.24 MB 2025-02-14 08:18:44,987 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32895.44 MB 2025-02-14 08:18:45,149 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7907] 2025-02-14 08:18:45,151 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:18:45,151 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 08:18:45,152 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:18:45,152 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 08:18:45,156 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 08:18:45,157 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:18:45,157 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 08:18:45,158 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 08:19:50,268 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:19:50,268 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 08:19:50,273 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 08:19:50,277 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:19:50,277 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 167, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 08:19:50,278 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:19:50,278 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 167, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 08:19:52,859 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 08:19:52,859 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 08:19:52,859 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.58 seconds 2025-02-14 08:19:52,859 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:19:52,859 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14132.39 MB 2025-02-14 08:19:52,859 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14723.39 MB 2025-02-14 08:19:52,859 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 591.00 MB 2025-02-14 08:19:52,859 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51359.25 MB 2025-02-14 08:19:52,859 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 16909.34 MB 2025-02-14 08:19:52,859 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34449.92 MB 2025-02-14 08:19:52,859 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23604.57 MB 2025-02-14 08:19:52,872 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 08:19:52,872 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 08:19:52,872 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:19:52,872 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:19:52,872 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14723.39 MB 2025-02-14 08:19:52,872 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14969.02 MB 2025-02-14 08:19:52,872 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 245.63 MB 2025-02-14 08:19:52,872 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 16909.34 MB 2025-02-14 08:19:52,872 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18327.01 MB 2025-02-14 08:19:52,872 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1417.67 MB 2025-02-14 08:19:52,872 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17012.31 MB 2025-02-14 08:19:53,641 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 08:19:53,641 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 08:19:53,641 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.77 seconds 2025-02-14 08:19:53,641 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:19:53,641 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14969.02 MB 2025-02-14 08:19:53,641 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15182.69 MB 2025-02-14 08:19:53,641 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 213.66 MB 2025-02-14 08:19:53,641 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18327.01 MB 2025-02-14 08:19:53,641 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17756.59 MB 2025-02-14 08:19:53,641 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -570.43 MB 2025-02-14 08:19:53,641 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19141.53 MB 2025-02-14 08:19:53,649 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 08:19:53,649 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 08:19:53,649 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 08:19:53,649 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:19:53,649 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15182.62 MB 2025-02-14 08:19:53,649 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15942.97 MB 2025-02-14 08:19:53,649 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 760.35 MB 2025-02-14 08:19:53,649 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17756.59 MB 2025-02-14 08:19:53,649 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17756.59 MB 2025-02-14 08:19:53,649 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:19:53,649 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16513.49 MB 2025-02-14 08:19:53,734 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 08:19:53,734 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 08:19:53,734 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 08:19:53,734 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:19:53,734 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15942.97 MB 2025-02-14 08:19:53,734 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16845.36 MB 2025-02-14 08:19:53,734 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 902.39 MB 2025-02-14 08:19:53,734 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17756.59 MB 2025-02-14 08:19:53,734 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20046.68 MB 2025-02-14 08:19:53,734 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2290.09 MB 2025-02-14 08:19:53,734 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19078.73 MB 2025-02-14 08:19:53,735 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 08:19:53,735 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 08:19:53,735 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 08:19:53,735 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:19:53,735 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15182.62 MB 2025-02-14 08:19:53,735 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16845.36 MB 2025-02-14 08:19:53,735 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1662.74 MB 2025-02-14 08:19:53,735 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17756.59 MB 2025-02-14 08:19:53,735 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20046.68 MB 2025-02-14 08:19:53,735 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2290.09 MB 2025-02-14 08:19:53,735 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19078.73 MB 2025-02-14 08:19:53,800 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 08:19:53,800 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 08:19:53,800 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 08:19:53,800 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:19:53,800 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17462.61 MB 2025-02-14 08:19:53,800 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17773.16 MB 2025-02-14 08:19:53,800 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 310.55 MB 2025-02-14 08:19:53,800 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20046.68 MB 2025-02-14 08:19:53,800 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20212.35 MB 2025-02-14 08:19:53,800 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 165.68 MB 2025-02-14 08:19:53,800 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18067.03 MB 2025-02-14 08:19:53,809 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 08:19:53,810 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 08:19:53,810 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:19:53,810 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:19:53,810 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17939.36 MB 2025-02-14 08:19:53,810 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18167.87 MB 2025-02-14 08:19:53,810 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.51 MB 2025-02-14 08:19:53,810 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20212.35 MB 2025-02-14 08:19:53,810 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20212.35 MB 2025-02-14 08:19:53,810 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:19:53,810 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18186.82 MB 2025-02-14 08:19:53,811 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 08:19:53,811 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 08:19:53,811 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.53 seconds 2025-02-14 08:19:53,811 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:19:53,811 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13550.55 MB 2025-02-14 08:19:53,811 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18368.92 MB 2025-02-14 08:19:53,811 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4818.37 MB 2025-02-14 08:19:53,811 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51359.25 MB 2025-02-14 08:19:53,811 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20212.35 MB 2025-02-14 08:19:53,811 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31146.90 MB 2025-02-14 08:19:53,811 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18368.92 MB 2025-02-14 08:19:54,077 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 08:19:54,077 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 08:19:54,077 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 08:19:54,077 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:19:54,077 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18368.92 MB 2025-02-14 08:19:54,077 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17428.48 MB 2025-02-14 08:19:54,077 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -940.44 MB 2025-02-14 08:19:54,077 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20212.35 MB 2025-02-14 08:19:54,077 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20212.35 MB 2025-02-14 08:19:54,077 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:19:54,077 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19172.55 MB 2025-02-14 08:19:54,097 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8161, cut from 8163 2025-02-14 08:19:54,097 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 08:19:54,103 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 08:19:54,103 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 08:19:54,103 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:19:54,103 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:19:54,103 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17428.48 MB 2025-02-14 08:19:54,103 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25867.32 MB 2025-02-14 08:19:54,103 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8438.84 MB 2025-02-14 08:19:54,104 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20212.35 MB 2025-02-14 08:19:54,104 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30698.11 MB 2025-02-14 08:19:54,104 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10485.76 MB 2025-02-14 08:19:54,104 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25867.32 MB 2025-02-14 08:19:54,254 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7953] 2025-02-14 08:19:54,256 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:19:54,256 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 08:19:54,257 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:19:54,257 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 08:19:54,261 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 08:19:54,262 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:19:54,262 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 08:19:54,262 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 08:21:27,947 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:21:27,948 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 08:21:27,955 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 08:21:27,962 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:21:27,962 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1520, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 08:21:27,963 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:21:27,964 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1520, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 08:21:51,265 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 08:21:51,265 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 08:21:51,265 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.29 seconds 2025-02-14 08:21:51,265 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:21:51,265 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23560.31 MB 2025-02-14 08:21:51,265 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28939.50 MB 2025-02-14 08:21:51,265 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5379.19 MB 2025-02-14 08:21:51,265 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39086.72 MB 2025-02-14 08:21:51,265 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39065.75 MB 2025-02-14 08:21:51,265 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20.97 MB 2025-02-14 08:21:51,265 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37788.83 MB 2025-02-14 08:21:51,346 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 08:21:51,346 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 08:21:51,346 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 08:21:51,346 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:21:51,346 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28939.50 MB 2025-02-14 08:21:51,346 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23679.86 MB 2025-02-14 08:21:51,346 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5259.65 MB 2025-02-14 08:21:51,346 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39065.75 MB 2025-02-14 08:21:51,346 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48085.60 MB 2025-02-14 08:21:51,346 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9019.85 MB 2025-02-14 08:21:51,346 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42560.03 MB 2025-02-14 08:21:53,255 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 08:21:53,255 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 08:21:53,255 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 08:21:53,255 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:21:53,255 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23679.86 MB 2025-02-14 08:21:53,255 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24210.70 MB 2025-02-14 08:21:53,255 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 08:21:53,255 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48085.60 MB 2025-02-14 08:21:53,255 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29492.25 MB 2025-02-14 08:21:53,255 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18593.35 MB 2025-02-14 08:21:53,255 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28190.03 MB 2025-02-14 08:21:53,270 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 08:21:53,270 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 08:21:53,270 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:21:53,270 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:21:53,270 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24210.70 MB 2025-02-14 08:21:53,270 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26100.23 MB 2025-02-14 08:21:53,270 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 08:21:53,270 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29492.25 MB 2025-02-14 08:21:53,270 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30435.97 MB 2025-02-14 08:21:53,270 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 08:21:53,270 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27517.66 MB 2025-02-14 08:21:53,479 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 08:21:53,479 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 08:21:53,479 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 08:21:53,479 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:21:53,479 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26100.23 MB 2025-02-14 08:21:53,479 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28342.09 MB 2025-02-14 08:21:53,479 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 08:21:53,479 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30435.97 MB 2025-02-14 08:21:53,479 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36098.28 MB 2025-02-14 08:21:53,479 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 08:21:53,479 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33886.37 MB 2025-02-14 08:21:53,479 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 08:21:53,479 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 08:21:53,480 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 08:21:53,480 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:21:53,480 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24210.70 MB 2025-02-14 08:21:53,480 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28342.09 MB 2025-02-14 08:21:53,480 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 08:21:53,480 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29492.25 MB 2025-02-14 08:21:53,480 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36098.28 MB 2025-02-14 08:21:53,480 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 08:21:53,480 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33886.37 MB 2025-02-14 08:21:53,649 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 08:21:53,649 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 08:21:53,649 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 08:21:53,649 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:21:53,649 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29875.63 MB 2025-02-14 08:21:53,649 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30642.63 MB 2025-02-14 08:21:53,649 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 08:21:53,649 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36098.28 MB 2025-02-14 08:21:53,650 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36513.51 MB 2025-02-14 08:21:53,650 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 08:21:53,650 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31350.42 MB 2025-02-14 08:21:53,668 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 08:21:53,669 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 08:21:53,669 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:21:53,669 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:21:53,669 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31055.52 MB 2025-02-14 08:21:53,669 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31283.97 MB 2025-02-14 08:21:53,669 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.44 MB 2025-02-14 08:21:53,669 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36513.51 MB 2025-02-14 08:21:53,669 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36513.51 MB 2025-02-14 08:21:53,669 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:21:53,669 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31491.36 MB 2025-02-14 08:21:53,670 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 08:21:53,670 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 08:21:53,670 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.70 seconds 2025-02-14 08:21:53,670 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:21:53,670 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18264.51 MB 2025-02-14 08:21:53,670 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31484.33 MB 2025-02-14 08:21:53,670 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13219.82 MB 2025-02-14 08:21:53,670 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39086.72 MB 2025-02-14 08:21:53,670 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36513.51 MB 2025-02-14 08:21:53,670 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2573.21 MB 2025-02-14 08:21:53,670 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31491.36 MB 2025-02-14 08:21:53,939 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 08:21:53,939 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 08:21:53,939 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 08:21:53,939 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:21:53,939 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31484.33 MB 2025-02-14 08:21:53,939 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23257.85 MB 2025-02-14 08:21:53,939 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8226.48 MB 2025-02-14 08:21:53,939 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36513.51 MB 2025-02-14 08:21:53,939 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36513.51 MB 2025-02-14 08:21:53,939 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:21:53,939 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33987.08 MB 2025-02-14 08:21:53,956 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8133, cut from 8135 2025-02-14 08:21:53,957 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 08:21:53,963 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 08:21:53,963 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 08:21:53,963 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:21:53,963 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:21:53,963 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23257.85 MB 2025-02-14 08:21:53,963 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31667.15 MB 2025-02-14 08:21:53,963 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8409.30 MB 2025-02-14 08:21:53,963 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36513.51 MB 2025-02-14 08:21:53,963 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44872.76 MB 2025-02-14 08:21:53,963 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8359.25 MB 2025-02-14 08:21:53,963 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31667.15 MB 2025-02-14 08:21:54,131 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7925] 2025-02-14 08:21:54,132 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:21:54,132 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 08:21:54,133 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:21:54,133 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 08:21:54,138 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 08:21:54,139 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:21:54,139 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 08:21:54,139 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 08:22:52,784 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:22:52,784 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 08:22:52,789 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 08:22:52,793 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:22:52,793 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1931, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 08:22:52,794 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:22:52,794 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1931, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 08:23:22,465 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 08:23:22,465 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 08:23:22,465 - resource_logging.py:150 - __exit__ - DEBUG - Time: 29.66 seconds 2025-02-14 08:23:22,465 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:23:22,465 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26424.22 MB 2025-02-14 08:23:22,465 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33258.84 MB 2025-02-14 08:23:22,465 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6834.62 MB 2025-02-14 08:23:22,465 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53232.01 MB 2025-02-14 08:23:22,465 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40464.55 MB 2025-02-14 08:23:22,465 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12767.46 MB 2025-02-14 08:23:22,465 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42238.19 MB 2025-02-14 08:23:22,585 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 08:23:22,585 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 08:23:22,585 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 08:23:22,585 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:23:22,585 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33258.84 MB 2025-02-14 08:23:22,585 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25816.52 MB 2025-02-14 08:23:22,585 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7442.32 MB 2025-02-14 08:23:22,585 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40464.55 MB 2025-02-14 08:23:22,585 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 62390.27 MB 2025-02-14 08:23:22,585 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 21925.72 MB 2025-02-14 08:23:22,585 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52798.16 MB 2025-02-14 08:23:24,519 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 08:23:24,519 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 08:23:24,519 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 08:23:24,519 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:23:24,519 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25816.52 MB 2025-02-14 08:23:24,519 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26347.36 MB 2025-02-14 08:23:24,519 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 08:23:24,519 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62390.27 MB 2025-02-14 08:23:24,519 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35045.51 MB 2025-02-14 08:23:24,519 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27344.76 MB 2025-02-14 08:23:24,519 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30326.69 MB 2025-02-14 08:23:24,532 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 08:23:24,532 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 08:23:24,532 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:23:24,532 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:23:24,532 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26347.36 MB 2025-02-14 08:23:24,532 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28236.89 MB 2025-02-14 08:23:24,532 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 08:23:24,532 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35045.51 MB 2025-02-14 08:23:24,532 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35045.51 MB 2025-02-14 08:23:24,532 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:23:24,532 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29654.32 MB 2025-02-14 08:23:24,745 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 08:23:24,745 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 08:23:24,745 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 08:23:24,745 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:23:24,745 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28236.89 MB 2025-02-14 08:23:24,745 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30478.75 MB 2025-02-14 08:23:24,745 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 08:23:24,745 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35045.51 MB 2025-02-14 08:23:24,745 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39764.10 MB 2025-02-14 08:23:24,745 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 08:23:24,745 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36023.03 MB 2025-02-14 08:23:24,746 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 08:23:24,746 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 08:23:24,746 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 08:23:24,746 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:23:24,746 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26347.36 MB 2025-02-14 08:23:24,746 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30478.75 MB 2025-02-14 08:23:24,746 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 08:23:24,746 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35045.51 MB 2025-02-14 08:23:24,746 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39764.10 MB 2025-02-14 08:23:24,746 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 08:23:24,746 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36023.03 MB 2025-02-14 08:23:24,949 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 08:23:24,949 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 08:23:24,949 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 08:23:24,949 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:23:24,949 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32012.29 MB 2025-02-14 08:23:24,949 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32779.29 MB 2025-02-14 08:23:24,949 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 08:23:24,949 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39764.10 MB 2025-02-14 08:23:24,949 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40177.24 MB 2025-02-14 08:23:24,949 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 08:23:24,949 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33487.08 MB 2025-02-14 08:23:24,973 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 08:23:24,973 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 08:23:24,973 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:23:24,973 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:23:24,973 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33192.18 MB 2025-02-14 08:23:24,973 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33420.26 MB 2025-02-14 08:23:24,973 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.08 MB 2025-02-14 08:23:24,973 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40177.24 MB 2025-02-14 08:23:24,973 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40177.24 MB 2025-02-14 08:23:24,973 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:23:24,973 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33645.09 MB 2025-02-14 08:23:24,974 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 08:23:24,974 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 08:23:24,974 - resource_logging.py:150 - __exit__ - DEBUG - Time: 32.18 seconds 2025-02-14 08:23:24,974 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:23:24,975 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19696.46 MB 2025-02-14 08:23:24,975 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33620.25 MB 2025-02-14 08:23:24,975 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13923.78 MB 2025-02-14 08:23:24,975 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53232.01 MB 2025-02-14 08:23:24,975 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40177.24 MB 2025-02-14 08:23:24,975 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13054.77 MB 2025-02-14 08:23:24,975 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33645.09 MB 2025-02-14 08:23:25,243 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 08:23:25,243 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 08:23:25,243 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 08:23:25,243 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:23:25,243 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33620.25 MB 2025-02-14 08:23:25,243 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24684.09 MB 2025-02-14 08:23:25,243 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8936.16 MB 2025-02-14 08:23:25,243 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40177.24 MB 2025-02-14 08:23:25,243 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40177.24 MB 2025-02-14 08:23:25,243 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:23:25,243 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36118.40 MB 2025-02-14 08:23:25,261 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8118, cut from 8120 2025-02-14 08:23:25,261 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 08:23:25,267 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 08:23:25,267 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 08:23:25,267 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:23:25,267 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:23:25,267 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24684.09 MB 2025-02-14 08:23:25,267 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33077.37 MB 2025-02-14 08:23:25,268 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8393.27 MB 2025-02-14 08:23:25,268 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40177.24 MB 2025-02-14 08:23:25,268 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48523.90 MB 2025-02-14 08:23:25,268 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8346.66 MB 2025-02-14 08:23:25,268 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33077.37 MB 2025-02-14 08:23:25,429 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7910] 2025-02-14 08:23:25,431 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:23:25,431 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 08:23:25,432 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:23:25,432 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 08:23:25,436 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 08:23:25,437 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:23:25,437 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 08:23:25,438 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 08:24:26,251 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:24:26,252 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 08:24:26,260 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 08:24:26,266 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:24:26,266 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1244, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 08:24:26,268 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:24:26,268 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1244, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 08:24:45,499 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 08:24:45,499 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 08:24:45,499 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.22 seconds 2025-02-14 08:24:45,499 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:24:45,499 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21637.10 MB 2025-02-14 08:24:45,499 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26039.54 MB 2025-02-14 08:24:45,499 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4402.45 MB 2025-02-14 08:24:45,499 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56870.57 MB 2025-02-14 08:24:45,499 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38010.88 MB 2025-02-14 08:24:45,499 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18859.69 MB 2025-02-14 08:24:45,499 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34958.84 MB 2025-02-14 08:24:45,572 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 08:24:45,572 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 08:24:45,572 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 08:24:45,572 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:24:45,572 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26039.54 MB 2025-02-14 08:24:45,572 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22245.02 MB 2025-02-14 08:24:45,572 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3794.53 MB 2025-02-14 08:24:45,572 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38010.88 MB 2025-02-14 08:24:45,572 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46651.15 MB 2025-02-14 08:24:45,572 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8640.27 MB 2025-02-14 08:24:45,572 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39075.22 MB 2025-02-14 08:24:47,480 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 08:24:47,480 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 08:24:47,480 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 08:24:47,480 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:24:47,480 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22245.02 MB 2025-02-14 08:24:47,480 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22775.86 MB 2025-02-14 08:24:47,480 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 08:24:47,480 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46651.15 MB 2025-02-14 08:24:47,480 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29433.53 MB 2025-02-14 08:24:47,480 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17217.62 MB 2025-02-14 08:24:47,480 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26755.19 MB 2025-02-14 08:24:47,493 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 08:24:47,493 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 08:24:47,493 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:24:47,493 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:24:47,493 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22775.86 MB 2025-02-14 08:24:47,493 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24665.39 MB 2025-02-14 08:24:47,493 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 08:24:47,493 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29433.53 MB 2025-02-14 08:24:47,493 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29433.53 MB 2025-02-14 08:24:47,493 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:24:47,493 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26082.82 MB 2025-02-14 08:24:47,703 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 08:24:47,703 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 08:24:47,703 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 08:24:47,703 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:24:47,703 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24665.39 MB 2025-02-14 08:24:47,703 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26907.25 MB 2025-02-14 08:24:47,703 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 08:24:47,703 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29433.53 MB 2025-02-14 08:24:47,703 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35095.84 MB 2025-02-14 08:24:47,703 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 08:24:47,703 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32451.53 MB 2025-02-14 08:24:47,703 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 08:24:47,703 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 08:24:47,703 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 08:24:47,704 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:24:47,704 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22775.86 MB 2025-02-14 08:24:47,704 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26907.25 MB 2025-02-14 08:24:47,704 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 08:24:47,704 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29433.53 MB 2025-02-14 08:24:47,704 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35095.84 MB 2025-02-14 08:24:47,704 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 08:24:47,704 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32451.53 MB 2025-02-14 08:24:47,873 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 08:24:47,873 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 08:24:47,873 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 08:24:47,873 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:24:47,873 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28440.79 MB 2025-02-14 08:24:47,873 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29207.79 MB 2025-02-14 08:24:47,873 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 08:24:47,873 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35095.84 MB 2025-02-14 08:24:47,873 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35513.17 MB 2025-02-14 08:24:47,873 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 08:24:47,873 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29915.58 MB 2025-02-14 08:24:47,892 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 08:24:47,892 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 08:24:47,892 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:24:47,892 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:24:47,892 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29620.68 MB 2025-02-14 08:24:47,892 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29849.69 MB 2025-02-14 08:24:47,892 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.01 MB 2025-02-14 08:24:47,892 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35513.17 MB 2025-02-14 08:24:47,892 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35513.17 MB 2025-02-14 08:24:47,892 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:24:47,892 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30089.04 MB 2025-02-14 08:24:47,893 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 08:24:47,893 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 08:24:47,893 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.62 seconds 2025-02-14 08:24:47,893 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:24:47,893 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17302.90 MB 2025-02-14 08:24:47,893 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30050.62 MB 2025-02-14 08:24:47,893 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12747.72 MB 2025-02-14 08:24:47,893 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56870.57 MB 2025-02-14 08:24:47,893 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35513.17 MB 2025-02-14 08:24:47,893 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21357.40 MB 2025-02-14 08:24:47,893 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30089.04 MB 2025-02-14 08:24:48,161 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 08:24:48,161 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 08:24:48,161 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 08:24:48,161 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:24:48,161 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30050.62 MB 2025-02-14 08:24:48,161 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22305.01 MB 2025-02-14 08:24:48,161 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7745.61 MB 2025-02-14 08:24:48,161 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35513.17 MB 2025-02-14 08:24:48,161 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35513.17 MB 2025-02-14 08:24:48,161 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:24:48,161 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32560.44 MB 2025-02-14 08:24:48,179 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8156, cut from 8158 2025-02-14 08:24:48,179 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 08:24:48,185 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 08:24:48,185 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 08:24:48,185 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:24:48,185 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:24:48,185 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22305.01 MB 2025-02-14 08:24:48,185 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30738.60 MB 2025-02-14 08:24:48,185 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8433.59 MB 2025-02-14 08:24:48,185 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35513.17 MB 2025-02-14 08:24:48,185 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43897.59 MB 2025-02-14 08:24:48,185 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8384.41 MB 2025-02-14 08:24:48,185 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30738.60 MB 2025-02-14 08:24:48,347 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7948] 2025-02-14 08:24:48,349 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:24:48,349 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 08:24:48,350 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:24:48,350 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 08:24:48,354 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 08:24:48,356 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:24:48,356 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 08:24:48,356 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 08:26:23,399 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:26:23,399 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 08:26:23,404 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 08:26:23,408 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:26:23,408 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1294, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 08:26:23,410 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:26:23,410 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1294, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 08:26:43,233 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 08:26:43,233 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 08:26:43,233 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.81 seconds 2025-02-14 08:26:43,233 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:26:43,233 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21985.51 MB 2025-02-14 08:26:43,233 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26565.69 MB 2025-02-14 08:26:43,233 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4580.18 MB 2025-02-14 08:26:43,233 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52282.00 MB 2025-02-14 08:26:43,233 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38208.01 MB 2025-02-14 08:26:43,233 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14073.99 MB 2025-02-14 08:26:43,233 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35533.74 MB 2025-02-14 08:26:43,306 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 08:26:43,306 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 08:26:43,306 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 08:26:43,307 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:26:43,307 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26565.69 MB 2025-02-14 08:26:43,307 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22504.95 MB 2025-02-14 08:26:43,307 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4060.73 MB 2025-02-14 08:26:43,307 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38208.01 MB 2025-02-14 08:26:43,307 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46953.14 MB 2025-02-14 08:26:43,307 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8745.12 MB 2025-02-14 08:26:43,307 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39765.27 MB 2025-02-14 08:26:45,220 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 08:26:45,220 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 08:26:45,220 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 08:26:45,220 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:26:45,220 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22504.95 MB 2025-02-14 08:26:45,220 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23035.79 MB 2025-02-14 08:26:45,220 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 08:26:45,220 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46953.14 MB 2025-02-14 08:26:45,220 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29454.50 MB 2025-02-14 08:26:45,220 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17498.64 MB 2025-02-14 08:26:45,220 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27015.13 MB 2025-02-14 08:26:45,233 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 08:26:45,233 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 08:26:45,233 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:26:45,233 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:26:45,233 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23035.79 MB 2025-02-14 08:26:45,233 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24925.33 MB 2025-02-14 08:26:45,233 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 08:26:45,233 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29454.50 MB 2025-02-14 08:26:45,233 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29454.50 MB 2025-02-14 08:26:45,233 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:26:45,233 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26342.76 MB 2025-02-14 08:26:45,446 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 08:26:45,446 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 08:26:45,446 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 08:26:45,446 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:26:45,446 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24925.33 MB 2025-02-14 08:26:45,446 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27167.18 MB 2025-02-14 08:26:45,446 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 08:26:45,446 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29454.50 MB 2025-02-14 08:26:45,446 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35116.81 MB 2025-02-14 08:26:45,446 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 08:26:45,446 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32711.47 MB 2025-02-14 08:26:45,446 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 08:26:45,447 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 08:26:45,447 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 08:26:45,447 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:26:45,447 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23035.79 MB 2025-02-14 08:26:45,447 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27167.18 MB 2025-02-14 08:26:45,447 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 08:26:45,447 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29454.50 MB 2025-02-14 08:26:45,447 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35116.81 MB 2025-02-14 08:26:45,447 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 08:26:45,447 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32711.47 MB 2025-02-14 08:26:45,614 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 08:26:45,614 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 08:26:45,614 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 08:26:45,614 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:26:45,614 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28700.73 MB 2025-02-14 08:26:45,614 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29467.73 MB 2025-02-14 08:26:45,614 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 08:26:45,614 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35116.81 MB 2025-02-14 08:26:45,614 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35532.05 MB 2025-02-14 08:26:45,614 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 08:26:45,614 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30175.52 MB 2025-02-14 08:26:45,633 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 08:26:45,633 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 08:26:45,633 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:26:45,633 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:26:45,633 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29880.62 MB 2025-02-14 08:26:45,633 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30108.72 MB 2025-02-14 08:26:45,633 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.10 MB 2025-02-14 08:26:45,633 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35532.05 MB 2025-02-14 08:26:45,633 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35532.05 MB 2025-02-14 08:26:45,633 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:26:45,633 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30315.56 MB 2025-02-14 08:26:45,634 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 08:26:45,634 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 08:26:45,634 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.22 seconds 2025-02-14 08:26:45,634 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:26:45,634 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17477.11 MB 2025-02-14 08:26:45,634 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30308.73 MB 2025-02-14 08:26:45,634 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12831.63 MB 2025-02-14 08:26:45,634 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52282.00 MB 2025-02-14 08:26:45,634 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35532.05 MB 2025-02-14 08:26:45,634 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16749.95 MB 2025-02-14 08:26:45,634 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30315.56 MB 2025-02-14 08:26:45,901 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 08:26:45,901 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 08:26:45,901 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 08:26:45,901 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:26:45,901 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30308.73 MB 2025-02-14 08:26:45,901 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22465.12 MB 2025-02-14 08:26:45,901 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7843.62 MB 2025-02-14 08:26:45,901 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35532.05 MB 2025-02-14 08:26:45,901 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35532.05 MB 2025-02-14 08:26:45,901 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:26:45,901 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32807.19 MB 2025-02-14 08:26:45,919 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8119, cut from 8121 2025-02-14 08:26:45,919 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 08:26:45,925 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 08:26:45,925 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 08:26:45,925 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:26:45,925 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:26:45,925 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22465.12 MB 2025-02-14 08:26:45,925 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30860.64 MB 2025-02-14 08:26:45,925 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8395.52 MB 2025-02-14 08:26:45,925 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35532.05 MB 2025-02-14 08:26:45,925 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39705.38 MB 2025-02-14 08:26:45,925 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4173.33 MB 2025-02-14 08:26:45,925 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30860.64 MB 2025-02-14 08:26:46,085 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7911] 2025-02-14 08:26:46,087 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:26:46,087 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 08:26:46,088 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:26:46,088 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 08:26:46,093 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 08:26:46,094 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:26:46,094 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 08:26:46,094 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 08:28:01,066 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:28:01,066 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 08:28:01,072 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 08:28:01,076 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:28:01,076 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1912, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 08:28:01,077 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:28:01,077 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1912, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 08:28:30,502 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 08:28:30,502 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 08:28:30,502 - resource_logging.py:150 - __exit__ - DEBUG - Time: 29.42 seconds 2025-02-14 08:28:30,502 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:28:30,502 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26291.83 MB 2025-02-14 08:28:30,502 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33059.34 MB 2025-02-14 08:28:30,502 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6767.51 MB 2025-02-14 08:28:30,502 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48052.04 MB 2025-02-14 08:28:30,502 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40370.18 MB 2025-02-14 08:28:30,502 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7681.87 MB 2025-02-14 08:28:30,502 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41879.30 MB 2025-02-14 08:28:30,612 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 08:28:30,612 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 08:28:30,612 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 08:28:30,612 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:28:30,612 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33059.34 MB 2025-02-14 08:28:30,612 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25717.74 MB 2025-02-14 08:28:30,612 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7341.60 MB 2025-02-14 08:28:30,612 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40370.18 MB 2025-02-14 08:28:30,612 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 59903.05 MB 2025-02-14 08:28:30,612 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 19532.87 MB 2025-02-14 08:28:30,612 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50908.33 MB 2025-02-14 08:28:32,542 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 08:28:32,542 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 08:28:32,542 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 08:28:32,542 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:28:32,542 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25717.74 MB 2025-02-14 08:28:32,542 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26248.58 MB 2025-02-14 08:28:32,542 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 08:28:32,542 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59903.05 MB 2025-02-14 08:28:32,542 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30844.91 MB 2025-02-14 08:28:32,542 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29058.14 MB 2025-02-14 08:28:32,542 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30228.96 MB 2025-02-14 08:28:32,556 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 08:28:32,556 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 08:28:32,556 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:28:32,556 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:28:32,556 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26248.58 MB 2025-02-14 08:28:32,556 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28138.12 MB 2025-02-14 08:28:32,556 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 08:28:32,556 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30844.91 MB 2025-02-14 08:28:32,556 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32732.35 MB 2025-02-14 08:28:32,556 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 08:28:32,556 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29555.55 MB 2025-02-14 08:28:32,772 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 08:28:32,772 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 08:28:32,772 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 08:28:32,772 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:28:32,772 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28138.12 MB 2025-02-14 08:28:32,772 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30379.97 MB 2025-02-14 08:28:32,772 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 08:28:32,772 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32732.35 MB 2025-02-14 08:28:32,772 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38866.52 MB 2025-02-14 08:28:32,772 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 08:28:32,772 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35924.26 MB 2025-02-14 08:28:32,773 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 08:28:32,773 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 08:28:32,773 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 08:28:32,773 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:28:32,773 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26248.58 MB 2025-02-14 08:28:32,773 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30379.97 MB 2025-02-14 08:28:32,773 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 08:28:32,773 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30844.91 MB 2025-02-14 08:28:32,773 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38866.52 MB 2025-02-14 08:28:32,773 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-14 08:28:32,773 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35924.26 MB 2025-02-14 08:28:32,941 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 08:28:32,941 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 08:28:32,942 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 08:28:32,942 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:28:32,942 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31913.52 MB 2025-02-14 08:28:32,942 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32680.52 MB 2025-02-14 08:28:32,942 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 08:28:32,942 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38866.52 MB 2025-02-14 08:28:32,942 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39281.75 MB 2025-02-14 08:28:32,942 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 08:28:32,942 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33388.31 MB 2025-02-14 08:28:32,960 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 08:28:32,960 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 08:28:32,960 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:28:32,960 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:28:32,960 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33093.41 MB 2025-02-14 08:28:32,960 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33321.43 MB 2025-02-14 08:28:32,960 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.03 MB 2025-02-14 08:28:32,960 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39281.75 MB 2025-02-14 08:28:32,960 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39281.75 MB 2025-02-14 08:28:32,960 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:28:32,960 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33540.65 MB 2025-02-14 08:28:32,961 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 08:28:32,961 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 08:28:32,961 - resource_logging.py:150 - __exit__ - DEBUG - Time: 31.88 seconds 2025-02-14 08:28:32,961 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:28:32,961 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19630.27 MB 2025-02-14 08:28:32,961 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33521.38 MB 2025-02-14 08:28:32,961 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13891.11 MB 2025-02-14 08:28:32,962 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48052.04 MB 2025-02-14 08:28:32,962 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39281.75 MB 2025-02-14 08:28:32,962 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8770.29 MB 2025-02-14 08:28:32,962 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33540.65 MB 2025-02-14 08:28:33,229 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 08:28:33,229 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 08:28:33,229 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 08:28:33,229 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:28:33,229 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33521.38 MB 2025-02-14 08:28:33,229 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24617.13 MB 2025-02-14 08:28:33,229 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8904.24 MB 2025-02-14 08:28:33,230 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39281.75 MB 2025-02-14 08:28:33,230 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39281.75 MB 2025-02-14 08:28:33,230 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:28:33,230 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36018.91 MB 2025-02-14 08:28:33,247 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8116, cut from 8118 2025-02-14 08:28:33,247 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 08:28:33,253 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 08:28:33,253 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 08:28:33,254 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:28:33,254 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:28:33,254 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24617.13 MB 2025-02-14 08:28:33,254 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33009.56 MB 2025-02-14 08:28:33,254 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8392.42 MB 2025-02-14 08:28:33,254 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39281.75 MB 2025-02-14 08:28:33,254 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47624.22 MB 2025-02-14 08:28:33,254 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8342.47 MB 2025-02-14 08:28:33,254 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33009.56 MB 2025-02-14 08:28:33,417 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7908] 2025-02-14 08:28:33,418 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:28:33,418 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 08:28:33,419 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:28:33,419 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 08:28:33,424 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 08:28:33,425 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:28:33,425 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 08:28:33,425 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 08:28:43,347 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:28:43,347 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 08:28:43,352 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 08:28:43,355 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:28:43,355 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1606, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 08:28:43,356 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:28:43,356 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1606, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 08:29:08,472 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 08:29:08,472 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 08:29:08,472 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.11 seconds 2025-02-14 08:29:08,472 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:29:08,472 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24159.57 MB 2025-02-14 08:29:08,472 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29843.12 MB 2025-02-14 08:29:08,472 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5683.54 MB 2025-02-14 08:29:08,472 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55966.70 MB 2025-02-14 08:29:08,472 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39281.75 MB 2025-02-14 08:29:08,472 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16684.94 MB 2025-02-14 08:29:08,472 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38841.08 MB 2025-02-14 08:29:08,563 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 08:29:08,563 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 08:29:08,563 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 08:29:08,563 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:29:08,563 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29843.12 MB 2025-02-14 08:29:08,563 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24126.94 MB 2025-02-14 08:29:08,563 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5716.17 MB 2025-02-14 08:29:08,563 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39281.75 MB 2025-02-14 08:29:08,563 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50774.15 MB 2025-02-14 08:29:08,563 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 11492.39 MB 2025-02-14 08:29:08,563 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45456.89 MB 2025-02-14 08:29:10,497 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 08:29:10,497 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 08:29:10,497 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 08:29:10,497 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:29:10,497 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24126.94 MB 2025-02-14 08:29:10,497 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24657.79 MB 2025-02-14 08:29:10,497 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 08:29:10,497 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50774.15 MB 2025-02-14 08:29:10,497 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29425.14 MB 2025-02-14 08:29:10,497 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21349.01 MB 2025-02-14 08:29:10,497 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28638.16 MB 2025-02-14 08:29:10,511 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 08:29:10,511 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 08:29:10,511 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:29:10,511 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:29:10,511 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24657.79 MB 2025-02-14 08:29:10,511 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26547.22 MB 2025-02-14 08:29:10,511 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.43 MB 2025-02-14 08:29:10,511 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29425.14 MB 2025-02-14 08:29:10,511 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30368.86 MB 2025-02-14 08:29:10,511 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 08:29:10,511 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27964.65 MB 2025-02-14 08:29:10,721 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 08:29:10,721 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 08:29:10,721 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 08:29:10,721 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:29:10,721 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26547.22 MB 2025-02-14 08:29:10,721 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28789.08 MB 2025-02-14 08:29:10,721 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 08:29:10,721 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30368.86 MB 2025-02-14 08:29:10,721 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36503.03 MB 2025-02-14 08:29:10,721 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 08:29:10,721 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34333.36 MB 2025-02-14 08:29:10,721 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 08:29:10,721 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 08:29:10,721 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 08:29:10,721 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:29:10,721 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24657.79 MB 2025-02-14 08:29:10,721 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28789.08 MB 2025-02-14 08:29:10,721 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.29 MB 2025-02-14 08:29:10,722 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29425.14 MB 2025-02-14 08:29:10,722 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36503.03 MB 2025-02-14 08:29:10,722 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7077.89 MB 2025-02-14 08:29:10,722 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34333.36 MB 2025-02-14 08:29:10,893 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 08:29:10,893 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 08:29:10,893 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 08:29:10,893 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:29:10,893 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30322.62 MB 2025-02-14 08:29:10,893 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31089.62 MB 2025-02-14 08:29:10,893 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 08:29:10,893 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36503.03 MB 2025-02-14 08:29:10,893 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36918.26 MB 2025-02-14 08:29:10,893 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 08:29:10,893 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31797.41 MB 2025-02-14 08:29:10,912 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 08:29:10,912 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 08:29:10,912 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:29:10,912 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:29:10,912 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31502.51 MB 2025-02-14 08:29:10,912 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31731.49 MB 2025-02-14 08:29:10,912 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.98 MB 2025-02-14 08:29:10,912 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36918.26 MB 2025-02-14 08:29:10,912 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36918.26 MB 2025-02-14 08:29:10,912 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:29:10,912 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31941.77 MB 2025-02-14 08:29:10,913 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 08:29:10,913 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 08:29:10,913 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.56 seconds 2025-02-14 08:29:10,913 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:29:10,913 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18564.14 MB 2025-02-14 08:29:10,914 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31932.49 MB 2025-02-14 08:29:10,914 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13368.35 MB 2025-02-14 08:29:10,914 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55966.70 MB 2025-02-14 08:29:10,914 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36918.26 MB 2025-02-14 08:29:10,914 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19048.43 MB 2025-02-14 08:29:10,914 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31941.77 MB 2025-02-14 08:29:11,199 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 08:29:11,199 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 08:29:11,199 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.28 seconds 2025-02-14 08:29:11,199 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:29:11,199 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31932.49 MB 2025-02-14 08:29:11,199 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23567.43 MB 2025-02-14 08:29:11,199 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8365.06 MB 2025-02-14 08:29:11,199 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36918.26 MB 2025-02-14 08:29:11,199 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36918.26 MB 2025-02-14 08:29:11,199 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:29:11,199 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34443.24 MB 2025-02-14 08:29:11,218 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8159, cut from 8161 2025-02-14 08:29:11,219 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 08:29:11,226 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 08:29:11,226 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 08:29:11,226 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:29:11,226 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:29:11,226 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23567.43 MB 2025-02-14 08:29:11,226 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32003.03 MB 2025-02-14 08:29:11,226 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8435.59 MB 2025-02-14 08:29:11,226 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36918.26 MB 2025-02-14 08:29:11,226 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45306.87 MB 2025-02-14 08:29:11,226 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8388.61 MB 2025-02-14 08:29:11,226 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32003.03 MB 2025-02-14 08:29:11,499 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7951] 2025-02-14 08:29:11,502 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:29:11,502 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 08:29:11,504 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:29:11,504 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 08:29:11,512 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 08:29:11,514 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:29:11,514 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 08:29:11,514 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 08:31:15,423 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:31:15,423 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 08:31:15,428 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 08:31:15,431 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:31:15,431 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 168, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 08:31:15,432 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:31:15,432 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 168, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 08:31:18,013 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 08:31:18,013 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 08:31:18,013 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.58 seconds 2025-02-14 08:31:18,014 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:31:18,014 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14139.36 MB 2025-02-14 08:31:18,014 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14733.83 MB 2025-02-14 08:31:18,014 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 594.48 MB 2025-02-14 08:31:18,014 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53695.48 MB 2025-02-14 08:31:18,014 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17383.29 MB 2025-02-14 08:31:18,014 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -36312.19 MB 2025-02-14 08:31:18,014 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23611.47 MB 2025-02-14 08:31:18,026 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 08:31:18,026 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 08:31:18,026 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:31:18,026 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:31:18,026 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14733.83 MB 2025-02-14 08:31:18,026 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14896.23 MB 2025-02-14 08:31:18,026 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 162.40 MB 2025-02-14 08:31:18,026 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17383.29 MB 2025-02-14 08:31:18,026 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18173.92 MB 2025-02-14 08:31:18,026 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 790.63 MB 2025-02-14 08:31:18,026 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16869.88 MB 2025-02-14 08:31:18,748 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 08:31:18,748 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 08:31:18,748 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.72 seconds 2025-02-14 08:31:18,748 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:31:18,748 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14896.23 MB 2025-02-14 08:31:18,748 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15095.30 MB 2025-02-14 08:31:18,748 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 199.07 MB 2025-02-14 08:31:18,748 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18173.92 MB 2025-02-14 08:31:18,748 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18104.71 MB 2025-02-14 08:31:18,748 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -69.21 MB 2025-02-14 08:31:18,748 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19067.71 MB 2025-02-14 08:31:18,755 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 08:31:18,755 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 08:31:18,755 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 08:31:18,755 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:31:18,755 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15095.30 MB 2025-02-14 08:31:18,755 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15803.70 MB 2025-02-14 08:31:18,755 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 708.40 MB 2025-02-14 08:31:18,755 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18104.71 MB 2025-02-14 08:31:18,755 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18104.71 MB 2025-02-14 08:31:18,755 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:31:18,755 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16335.24 MB 2025-02-14 08:31:18,837 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 08:31:18,837 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 08:31:18,837 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 08:31:18,837 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:31:18,837 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15803.70 MB 2025-02-14 08:31:18,837 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16644.44 MB 2025-02-14 08:31:18,837 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 840.74 MB 2025-02-14 08:31:18,837 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18104.71 MB 2025-02-14 08:31:18,837 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19700.65 MB 2025-02-14 08:31:18,837 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1595.93 MB 2025-02-14 08:31:18,837 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18725.60 MB 2025-02-14 08:31:18,838 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 08:31:18,838 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 08:31:18,838 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 08:31:18,838 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:31:18,838 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15095.30 MB 2025-02-14 08:31:18,838 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16644.44 MB 2025-02-14 08:31:18,838 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1549.14 MB 2025-02-14 08:31:18,838 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18104.71 MB 2025-02-14 08:31:18,838 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19700.65 MB 2025-02-14 08:31:18,838 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1595.93 MB 2025-02-14 08:31:18,838 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18725.60 MB 2025-02-14 08:31:18,902 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 08:31:18,902 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 08:31:18,902 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 08:31:18,902 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:31:18,902 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17219.52 MB 2025-02-14 08:31:18,902 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17507.14 MB 2025-02-14 08:31:18,902 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 287.63 MB 2025-02-14 08:31:18,902 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19700.65 MB 2025-02-14 08:31:18,902 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19851.64 MB 2025-02-14 08:31:18,902 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 150.99 MB 2025-02-14 08:31:18,902 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17782.81 MB 2025-02-14 08:31:18,913 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 08:31:18,913 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 08:31:18,913 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:31:18,913 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:31:18,913 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17661.98 MB 2025-02-14 08:31:18,913 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17881.29 MB 2025-02-14 08:31:18,913 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 219.31 MB 2025-02-14 08:31:18,913 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19851.64 MB 2025-02-14 08:31:18,913 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19851.64 MB 2025-02-14 08:31:18,913 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:31:18,913 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17891.72 MB 2025-02-14 08:31:18,915 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 08:31:18,915 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 08:31:18,915 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.48 seconds 2025-02-14 08:31:18,915 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:31:18,915 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13554.03 MB 2025-02-14 08:31:18,915 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18082.22 MB 2025-02-14 08:31:18,915 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4528.19 MB 2025-02-14 08:31:18,915 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53695.48 MB 2025-02-14 08:31:18,915 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19851.64 MB 2025-02-14 08:31:18,915 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33843.84 MB 2025-02-14 08:31:18,915 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18082.22 MB 2025-02-14 08:31:19,179 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 08:31:19,179 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 08:31:19,179 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 08:31:19,179 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:31:19,179 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14364.23 MB 2025-02-14 08:31:19,179 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17376.05 MB 2025-02-14 08:31:19,179 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3011.82 MB 2025-02-14 08:31:19,179 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19851.64 MB 2025-02-14 08:31:19,179 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19851.64 MB 2025-02-14 08:31:19,179 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:31:19,179 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17677.20 MB 2025-02-14 08:31:19,197 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8156, cut from 8158 2025-02-14 08:31:19,197 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 08:31:19,203 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 08:31:19,204 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 08:31:19,204 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:31:19,204 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:31:19,204 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17376.05 MB 2025-02-14 08:31:19,204 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25809.35 MB 2025-02-14 08:31:19,204 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8433.30 MB 2025-02-14 08:31:19,204 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19851.64 MB 2025-02-14 08:31:19,204 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30333.21 MB 2025-02-14 08:31:19,204 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10481.57 MB 2025-02-14 08:31:19,204 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25809.35 MB 2025-02-14 08:31:19,365 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7948] 2025-02-14 08:31:19,367 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:31:19,367 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 08:31:19,368 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:31:19,368 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 08:31:19,373 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 08:31:19,374 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:31:19,374 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 08:31:19,374 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 08:33:28,287 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:33:28,287 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 08:33:28,292 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 08:33:28,296 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:33:28,296 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2879, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 08:33:28,298 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:33:28,298 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2879, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 08:34:12,394 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 08:34:12,394 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 08:34:12,394 - resource_logging.py:150 - __exit__ - DEBUG - Time: 44.08 seconds 2025-02-14 08:34:12,394 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:34:12,394 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33030.72 MB 2025-02-14 08:34:12,394 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43219.34 MB 2025-02-14 08:34:12,394 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 10188.62 MB 2025-02-14 08:34:12,394 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58783.17 MB 2025-02-14 08:34:12,394 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47164.95 MB 2025-02-14 08:34:12,394 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11618.22 MB 2025-02-14 08:34:12,394 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 53407.96 MB 2025-02-14 08:34:12,562 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 08:34:12,562 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 08:34:12,562 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 08:34:12,562 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:34:12,562 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43219.34 MB 2025-02-14 08:34:12,562 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30745.55 MB 2025-02-14 08:34:12,562 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -12473.79 MB 2025-02-14 08:34:12,562 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47164.95 MB 2025-02-14 08:34:12,562 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 79668.71 MB 2025-02-14 08:34:12,562 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 32503.76 MB 2025-02-14 08:34:12,562 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 68218.35 MB 2025-02-14 08:34:14,548 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 08:34:14,548 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 08:34:14,548 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.98 seconds 2025-02-14 08:34:14,548 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:34:14,548 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30745.55 MB 2025-02-14 08:34:14,548 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31276.39 MB 2025-02-14 08:34:14,548 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 08:34:14,548 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 79668.71 MB 2025-02-14 08:34:14,548 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33294.39 MB 2025-02-14 08:34:14,548 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -46374.32 MB 2025-02-14 08:34:14,548 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35256.77 MB 2025-02-14 08:34:14,562 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 08:34:14,562 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 08:34:14,562 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:34:14,562 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:34:14,562 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31276.39 MB 2025-02-14 08:34:14,562 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33165.60 MB 2025-02-14 08:34:14,562 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.21 MB 2025-02-14 08:34:14,562 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33294.39 MB 2025-02-14 08:34:14,562 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36597.40 MB 2025-02-14 08:34:14,562 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 08:34:14,562 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34583.03 MB 2025-02-14 08:34:14,771 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 08:34:14,771 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 08:34:14,771 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 08:34:14,771 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:34:14,771 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33165.60 MB 2025-02-14 08:34:14,771 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35407.46 MB 2025-02-14 08:34:14,771 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 08:34:14,771 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36597.40 MB 2025-02-14 08:34:14,771 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43203.43 MB 2025-02-14 08:34:14,771 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 08:34:14,771 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40951.74 MB 2025-02-14 08:34:14,772 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 08:34:14,772 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 08:34:14,772 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 08:34:14,772 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:34:14,772 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31276.39 MB 2025-02-14 08:34:14,772 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35407.46 MB 2025-02-14 08:34:14,772 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.06 MB 2025-02-14 08:34:14,772 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33294.39 MB 2025-02-14 08:34:14,772 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43203.43 MB 2025-02-14 08:34:14,772 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9909.04 MB 2025-02-14 08:34:14,772 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40951.74 MB 2025-02-14 08:34:14,938 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 08:34:14,939 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 08:34:14,939 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 08:34:14,939 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:34:14,939 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36941.00 MB 2025-02-14 08:34:14,939 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37708.00 MB 2025-02-14 08:34:14,939 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 08:34:14,939 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43203.43 MB 2025-02-14 08:34:14,939 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43618.66 MB 2025-02-14 08:34:14,939 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 08:34:14,939 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38415.79 MB 2025-02-14 08:34:14,957 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 08:34:14,957 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 08:34:14,957 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:34:14,957 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:34:14,957 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38120.89 MB 2025-02-14 08:34:14,957 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38350.43 MB 2025-02-14 08:34:14,957 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.54 MB 2025-02-14 08:34:14,957 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43618.66 MB 2025-02-14 08:34:14,957 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43618.66 MB 2025-02-14 08:34:14,957 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:34:14,957 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38569.85 MB 2025-02-14 08:34:14,958 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 08:34:14,958 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 08:34:14,958 - resource_logging.py:150 - __exit__ - DEBUG - Time: 46.66 seconds 2025-02-14 08:34:14,958 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:34:14,958 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22999.71 MB 2025-02-14 08:34:14,958 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38551.50 MB 2025-02-14 08:34:14,958 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 15551.79 MB 2025-02-14 08:34:14,959 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48750.40 MB 2025-02-14 08:34:14,959 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43618.66 MB 2025-02-14 08:34:14,959 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5131.73 MB 2025-02-14 08:34:14,959 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38569.85 MB 2025-02-14 08:34:15,229 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 08:34:15,229 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 08:34:15,229 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 08:34:15,229 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:34:15,229 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38551.50 MB 2025-02-14 08:34:15,229 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28004.10 MB 2025-02-14 08:34:15,229 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10547.40 MB 2025-02-14 08:34:15,229 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43618.66 MB 2025-02-14 08:34:15,229 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43618.66 MB 2025-02-14 08:34:15,229 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:34:15,229 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40359.90 MB 2025-02-14 08:34:15,246 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 08:34:15,247 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 08:34:15,252 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 08:34:15,252 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 08:34:15,252 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:34:15,252 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:34:15,252 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28004.10 MB 2025-02-14 08:34:15,252 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36442.79 MB 2025-02-14 08:34:15,253 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8438.69 MB 2025-02-14 08:34:15,253 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43618.66 MB 2025-02-14 08:34:15,253 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47815.07 MB 2025-02-14 08:34:15,253 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4196.40 MB 2025-02-14 08:34:15,253 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36442.79 MB 2025-02-14 08:34:15,417 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 08:34:15,418 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:34:15,418 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 08:34:15,419 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:34:15,419 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 08:34:15,424 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 08:34:15,425 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:34:15,425 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 08:34:15,425 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 08:34:28,593 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:34:28,593 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 08:34:28,598 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 08:34:28,601 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:34:28,601 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 3383, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 08:34:28,602 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:34:28,602 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 3383, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 08:35:21,572 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 08:35:21,572 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 08:35:21,572 - resource_logging.py:150 - __exit__ - DEBUG - Time: 52.96 seconds 2025-02-14 08:35:21,572 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:35:21,572 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36542.40 MB 2025-02-14 08:35:21,572 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48515.04 MB 2025-02-14 08:35:21,572 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11972.64 MB 2025-02-14 08:35:21,572 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 79779.86 MB 2025-02-14 08:35:21,572 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52460.26 MB 2025-02-14 08:35:21,572 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27319.60 MB 2025-02-14 08:35:21,572 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 60487.29 MB 2025-02-14 08:35:21,743 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 08:35:21,743 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 08:35:21,743 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 08:35:21,743 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:35:21,743 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 48515.04 MB 2025-02-14 08:35:21,743 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33365.42 MB 2025-02-14 08:35:21,743 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -15149.62 MB 2025-02-14 08:35:21,743 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52460.26 MB 2025-02-14 08:35:21,743 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 78462.84 MB 2025-02-14 08:35:21,743 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 26002.59 MB 2025-02-14 08:35:21,743 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 68673.29 MB 2025-02-14 08:35:23,724 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 08:35:23,724 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 08:35:23,724 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.98 seconds 2025-02-14 08:35:23,724 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:35:23,724 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33365.42 MB 2025-02-14 08:35:23,724 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33896.26 MB 2025-02-14 08:35:23,724 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 08:35:23,724 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 78462.84 MB 2025-02-14 08:35:23,724 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35915.83 MB 2025-02-14 08:35:23,724 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -42547.02 MB 2025-02-14 08:35:23,724 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37876.63 MB 2025-02-14 08:35:23,738 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 08:35:23,738 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 08:35:23,738 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:35:23,738 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:35:23,738 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33896.26 MB 2025-02-14 08:35:23,738 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35785.79 MB 2025-02-14 08:35:23,738 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 08:35:23,738 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35915.83 MB 2025-02-14 08:35:23,738 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39218.84 MB 2025-02-14 08:35:23,738 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 08:35:23,738 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37203.22 MB 2025-02-14 08:35:23,946 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 08:35:23,946 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 08:35:23,947 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 08:35:23,947 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:35:23,947 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35785.79 MB 2025-02-14 08:35:23,947 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38027.65 MB 2025-02-14 08:35:23,947 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 08:35:23,947 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39218.84 MB 2025-02-14 08:35:23,947 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45824.87 MB 2025-02-14 08:35:23,947 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 08:35:23,947 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43571.93 MB 2025-02-14 08:35:23,947 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 08:35:23,947 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 08:35:23,947 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 08:35:23,947 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:35:23,947 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33896.26 MB 2025-02-14 08:35:23,947 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38027.65 MB 2025-02-14 08:35:23,947 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 08:35:23,947 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35915.83 MB 2025-02-14 08:35:23,947 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45824.87 MB 2025-02-14 08:35:23,947 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9909.04 MB 2025-02-14 08:35:23,947 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43571.93 MB 2025-02-14 08:35:24,131 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 08:35:24,131 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 08:35:24,131 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-14 08:35:24,131 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:35:24,131 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39561.19 MB 2025-02-14 08:35:24,131 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40328.19 MB 2025-02-14 08:35:24,131 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 08:35:24,131 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45824.87 MB 2025-02-14 08:35:24,131 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46238.01 MB 2025-02-14 08:35:24,131 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 08:35:24,131 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41035.98 MB 2025-02-14 08:35:24,150 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 08:35:24,150 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 08:35:24,150 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:35:24,150 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:35:24,150 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40741.08 MB 2025-02-14 08:35:24,150 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40970.18 MB 2025-02-14 08:35:24,150 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.10 MB 2025-02-14 08:35:24,150 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46238.01 MB 2025-02-14 08:35:24,150 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46238.01 MB 2025-02-14 08:35:24,150 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:35:24,150 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41213.40 MB 2025-02-14 08:35:24,151 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 08:35:24,151 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 08:35:24,151 - resource_logging.py:150 - __exit__ - DEBUG - Time: 55.55 seconds 2025-02-14 08:35:24,151 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:35:24,151 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24755.55 MB 2025-02-14 08:35:24,151 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41170.96 MB 2025-02-14 08:35:24,151 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 16415.41 MB 2025-02-14 08:35:24,151 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 67991.76 MB 2025-02-14 08:35:24,151 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46238.01 MB 2025-02-14 08:35:24,151 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21753.76 MB 2025-02-14 08:35:24,151 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41213.40 MB 2025-02-14 08:35:24,421 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 08:35:24,421 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 08:35:24,421 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 08:35:24,421 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:35:24,421 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41170.96 MB 2025-02-14 08:35:24,421 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29755.37 MB 2025-02-14 08:35:24,421 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -11415.59 MB 2025-02-14 08:35:24,421 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46238.01 MB 2025-02-14 08:35:24,421 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46238.01 MB 2025-02-14 08:35:24,421 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:35:24,421 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43678.94 MB 2025-02-14 08:35:24,439 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8150, cut from 8152 2025-02-14 08:35:24,439 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 08:35:24,445 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 08:35:24,445 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 08:35:24,445 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:35:24,445 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:35:24,445 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29755.37 MB 2025-02-14 08:35:24,445 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38181.67 MB 2025-02-14 08:35:24,445 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8426.30 MB 2025-02-14 08:35:24,445 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46238.01 MB 2025-02-14 08:35:24,445 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50428.12 MB 2025-02-14 08:35:24,445 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4190.11 MB 2025-02-14 08:35:24,445 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38181.67 MB 2025-02-14 08:35:24,608 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7942] 2025-02-14 08:35:24,609 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:35:24,609 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 08:35:24,610 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:35:24,610 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 08:35:24,615 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 08:35:24,616 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:35:24,616 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 08:35:24,616 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 08:36:12,228 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:36:12,228 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 08:36:12,233 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 08:36:12,237 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:36:12,237 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 235, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 08:36:12,238 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:36:12,238 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 235, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 08:36:15,894 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 08:36:15,894 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 08:36:15,894 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.65 seconds 2025-02-14 08:36:15,894 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:36:15,894 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14606.22 MB 2025-02-14 08:36:15,894 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15437.88 MB 2025-02-14 08:36:15,894 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 831.65 MB 2025-02-14 08:36:15,894 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58804.14 MB 2025-02-14 08:36:15,894 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18324.91 MB 2025-02-14 08:36:15,894 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -40479.23 MB 2025-02-14 08:36:15,894 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24304.89 MB 2025-02-14 08:36:15,912 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 08:36:15,912 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 08:36:15,912 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:36:15,912 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:36:15,912 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15437.88 MB 2025-02-14 08:36:15,912 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15841.66 MB 2025-02-14 08:36:15,912 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 403.79 MB 2025-02-14 08:36:15,912 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18324.91 MB 2025-02-14 08:36:15,912 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20394.80 MB 2025-02-14 08:36:15,912 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2069.89 MB 2025-02-14 08:36:15,912 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18779.46 MB 2025-02-14 08:36:17,049 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 08:36:17,049 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 08:36:17,049 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.14 seconds 2025-02-14 08:36:17,049 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:36:17,049 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15841.66 MB 2025-02-14 08:36:17,049 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16153.53 MB 2025-02-14 08:36:17,049 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 311.87 MB 2025-02-14 08:36:17,049 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20394.80 MB 2025-02-14 08:36:17,049 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19088.28 MB 2025-02-14 08:36:17,049 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1306.53 MB 2025-02-14 08:36:17,049 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20098.13 MB 2025-02-14 08:36:17,059 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 08:36:17,059 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 08:36:17,059 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:36:17,059 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:36:17,059 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16153.53 MB 2025-02-14 08:36:17,059 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17263.36 MB 2025-02-14 08:36:17,059 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1109.83 MB 2025-02-14 08:36:17,059 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19088.28 MB 2025-02-14 08:36:17,059 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19644.02 MB 2025-02-14 08:36:17,059 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 555.75 MB 2025-02-14 08:36:17,059 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18096.11 MB 2025-02-14 08:36:17,187 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 08:36:17,187 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 08:36:17,187 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 08:36:17,187 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:36:17,187 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17263.36 MB 2025-02-14 08:36:17,187 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18580.48 MB 2025-02-14 08:36:17,187 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1317.12 MB 2025-02-14 08:36:17,187 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19644.02 MB 2025-02-14 08:36:17,187 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23536.34 MB 2025-02-14 08:36:17,187 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3892.31 MB 2025-02-14 08:36:17,187 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21837.72 MB 2025-02-14 08:36:17,188 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 08:36:17,188 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 08:36:17,188 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-14 08:36:17,188 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:36:17,188 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16153.53 MB 2025-02-14 08:36:17,188 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18580.48 MB 2025-02-14 08:36:17,188 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2426.95 MB 2025-02-14 08:36:17,188 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19088.28 MB 2025-02-14 08:36:17,188 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23536.34 MB 2025-02-14 08:36:17,188 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4448.06 MB 2025-02-14 08:36:17,188 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21837.72 MB 2025-02-14 08:36:17,327 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 08:36:17,327 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 08:36:17,327 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 08:36:17,327 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:36:17,327 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19481.44 MB 2025-02-14 08:36:17,327 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19932.05 MB 2025-02-14 08:36:17,327 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 450.61 MB 2025-02-14 08:36:17,327 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23536.34 MB 2025-02-14 08:36:17,327 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23779.61 MB 2025-02-14 08:36:17,327 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 243.27 MB 2025-02-14 08:36:17,327 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20347.88 MB 2025-02-14 08:36:17,345 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 08:36:17,345 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 08:36:17,345 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:36:17,345 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:36:17,345 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20174.63 MB 2025-02-14 08:36:17,345 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20382.76 MB 2025-02-14 08:36:17,346 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 208.13 MB 2025-02-14 08:36:17,346 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23779.61 MB 2025-02-14 08:36:17,346 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23779.61 MB 2025-02-14 08:36:17,346 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:36:17,346 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20449.21 MB 2025-02-14 08:36:17,347 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 08:36:17,347 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 08:36:17,347 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.11 seconds 2025-02-14 08:36:17,348 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:36:17,348 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13787.47 MB 2025-02-14 08:36:17,348 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20583.83 MB 2025-02-14 08:36:17,348 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6796.36 MB 2025-02-14 08:36:17,348 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58804.14 MB 2025-02-14 08:36:17,348 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23779.61 MB 2025-02-14 08:36:17,348 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35024.54 MB 2025-02-14 08:36:17,348 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20583.83 MB 2025-02-14 08:36:17,638 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 08:36:17,638 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 08:36:17,638 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-14 08:36:17,638 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:36:17,638 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20583.83 MB 2025-02-14 08:36:17,638 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23597.86 MB 2025-02-14 08:36:17,638 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-14 08:36:17,638 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23779.61 MB 2025-02-14 08:36:17,638 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25390.22 MB 2025-02-14 08:36:17,639 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1610.61 MB 2025-02-14 08:36:17,639 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23899.49 MB 2025-02-14 08:36:17,658 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 08:36:17,659 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 08:36:17,666 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 08:36:17,666 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 08:36:17,666 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:36:17,666 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:36:17,666 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18012.78 MB 2025-02-14 08:36:17,666 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26451.80 MB 2025-02-14 08:36:17,666 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 08:36:17,666 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25390.22 MB 2025-02-14 08:36:17,666 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35880.17 MB 2025-02-14 08:36:17,666 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 08:36:17,666 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26451.80 MB 2025-02-14 08:36:17,925 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 08:36:17,927 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:36:17,927 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 08:36:17,929 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:36:17,929 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 08:36:17,937 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 08:36:17,939 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:36:17,939 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 08:36:17,939 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 08:36:34,425 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:36:34,425 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 08:36:34,430 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 08:36:34,433 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:36:34,433 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1088, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 08:36:34,434 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:36:34,434 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1088, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 08:36:51,273 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 08:36:51,273 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 08:36:51,273 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.83 seconds 2025-02-14 08:36:51,273 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:36:51,273 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20550.06 MB 2025-02-14 08:36:51,273 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24400.44 MB 2025-02-14 08:36:51,273 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3850.37 MB 2025-02-14 08:36:51,273 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48465.18 MB 2025-02-14 08:36:51,273 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31247.56 MB 2025-02-14 08:36:51,273 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17217.62 MB 2025-02-14 08:36:51,273 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33418.82 MB 2025-02-14 08:36:51,334 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 08:36:51,334 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 08:36:51,334 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 08:36:51,334 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:36:51,335 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24400.44 MB 2025-02-14 08:36:51,335 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21434.02 MB 2025-02-14 08:36:51,335 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2966.41 MB 2025-02-14 08:36:51,335 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31247.56 MB 2025-02-14 08:36:51,335 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39862.67 MB 2025-02-14 08:36:51,335 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8615.10 MB 2025-02-14 08:36:51,335 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35622.63 MB 2025-02-14 08:36:53,262 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 08:36:53,262 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 08:36:53,262 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 08:36:53,262 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:36:53,262 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21434.02 MB 2025-02-14 08:36:53,262 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21964.86 MB 2025-02-14 08:36:53,262 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 08:36:53,262 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39862.67 MB 2025-02-14 08:36:53,262 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28812.77 MB 2025-02-14 08:36:53,262 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11049.89 MB 2025-02-14 08:36:53,262 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25944.20 MB 2025-02-14 08:36:53,275 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 08:36:53,275 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 08:36:53,275 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:36:53,275 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:36:53,275 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21964.86 MB 2025-02-14 08:36:53,275 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23854.40 MB 2025-02-14 08:36:53,275 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 08:36:53,275 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28812.77 MB 2025-02-14 08:36:53,275 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28812.77 MB 2025-02-14 08:36:53,275 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:36:53,275 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25271.83 MB 2025-02-14 08:36:53,485 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 08:36:53,485 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 08:36:53,485 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 08:36:53,485 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:36:53,485 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23854.40 MB 2025-02-14 08:36:53,485 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26096.25 MB 2025-02-14 08:36:53,485 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 08:36:53,485 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28812.77 MB 2025-02-14 08:36:53,485 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34475.08 MB 2025-02-14 08:36:53,485 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 08:36:53,485 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31640.54 MB 2025-02-14 08:36:53,486 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 08:36:53,486 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 08:36:53,486 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 08:36:53,486 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:36:53,486 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21964.86 MB 2025-02-14 08:36:53,486 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26096.25 MB 2025-02-14 08:36:53,486 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 08:36:53,486 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28812.77 MB 2025-02-14 08:36:53,486 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34475.08 MB 2025-02-14 08:36:53,486 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 08:36:53,486 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31640.54 MB 2025-02-14 08:36:53,653 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 08:36:53,653 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 08:36:53,653 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 08:36:53,653 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:36:53,653 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27629.80 MB 2025-02-14 08:36:53,653 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28396.80 MB 2025-02-14 08:36:53,653 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 08:36:53,653 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34475.08 MB 2025-02-14 08:36:53,653 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34890.32 MB 2025-02-14 08:36:53,653 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 08:36:53,653 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29104.59 MB 2025-02-14 08:36:53,672 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 08:36:53,672 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 08:36:53,672 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:36:53,672 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:36:53,672 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28809.69 MB 2025-02-14 08:36:53,672 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29038.86 MB 2025-02-14 08:36:53,672 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.17 MB 2025-02-14 08:36:53,672 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34890.32 MB 2025-02-14 08:36:53,672 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34890.32 MB 2025-02-14 08:36:53,672 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:36:53,672 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29236.41 MB 2025-02-14 08:36:53,673 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 08:36:53,673 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 08:36:53,673 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.24 seconds 2025-02-14 08:36:53,673 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:36:53,673 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16759.39 MB 2025-02-14 08:36:53,673 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29239.93 MB 2025-02-14 08:36:53,673 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12480.55 MB 2025-02-14 08:36:53,673 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48465.18 MB 2025-02-14 08:36:53,673 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34890.32 MB 2025-02-14 08:36:53,673 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13574.86 MB 2025-02-14 08:36:53,673 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29239.93 MB 2025-02-14 08:36:53,942 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 08:36:53,942 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 08:36:53,942 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 08:36:53,942 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:36:53,942 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29239.93 MB 2025-02-14 08:36:53,942 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21763.77 MB 2025-02-14 08:36:53,942 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7476.16 MB 2025-02-14 08:36:53,942 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34890.32 MB 2025-02-14 08:36:53,942 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34890.32 MB 2025-02-14 08:36:53,942 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:36:53,942 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31751.60 MB 2025-02-14 08:36:53,960 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 08:36:53,960 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 08:36:53,966 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 08:36:53,966 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 08:36:53,966 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:36:53,966 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:36:53,966 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21763.77 MB 2025-02-14 08:36:53,966 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30202.80 MB 2025-02-14 08:36:53,966 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 08:36:53,966 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34890.32 MB 2025-02-14 08:36:53,966 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43281.02 MB 2025-02-14 08:36:53,966 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 08:36:53,966 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30202.80 MB 2025-02-14 08:36:54,129 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 08:36:54,131 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:36:54,131 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 08:36:54,132 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:36:54,132 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 08:36:54,136 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 08:36:54,137 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:36:54,137 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 08:36:54,138 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 08:37:14,148 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:37:14,148 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 08:37:14,153 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 08:37:14,157 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:37:14,157 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 366, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 08:37:14,158 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:37:14,158 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 366, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 08:37:19,844 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 08:37:19,844 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 08:37:19,844 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.68 seconds 2025-02-14 08:37:19,844 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:37:19,844 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15519.05 MB 2025-02-14 08:37:19,844 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16815.09 MB 2025-02-14 08:37:19,844 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1296.04 MB 2025-02-14 08:37:19,844 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55866.03 MB 2025-02-14 08:37:19,844 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20092.81 MB 2025-02-14 08:37:19,844 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35773.22 MB 2025-02-14 08:37:19,844 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25670.71 MB 2025-02-14 08:37:19,866 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 08:37:19,866 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 08:37:19,866 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:37:19,866 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:37:19,866 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16815.09 MB 2025-02-14 08:37:19,866 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17161.84 MB 2025-02-14 08:37:19,866 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 346.75 MB 2025-02-14 08:37:19,866 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20092.81 MB 2025-02-14 08:37:19,866 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24102.57 MB 2025-02-14 08:37:19,866 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4009.75 MB 2025-02-14 08:37:19,866 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21445.60 MB 2025-02-14 08:37:21,444 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 08:37:21,444 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 08:37:21,444 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.58 seconds 2025-02-14 08:37:21,444 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:37:21,444 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17161.84 MB 2025-02-14 08:37:21,444 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17594.48 MB 2025-02-14 08:37:21,444 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 432.64 MB 2025-02-14 08:37:21,444 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24102.57 MB 2025-02-14 08:37:21,444 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21091.06 MB 2025-02-14 08:37:21,444 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3011.51 MB 2025-02-14 08:37:21,444 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21588.12 MB 2025-02-14 08:37:21,456 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 08:37:21,456 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 08:37:21,456 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:37:21,456 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:37:21,456 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17594.48 MB 2025-02-14 08:37:21,456 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19135.62 MB 2025-02-14 08:37:21,456 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1541.14 MB 2025-02-14 08:37:21,456 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21091.06 MB 2025-02-14 08:37:21,456 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22630.37 MB 2025-02-14 08:37:21,456 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1539.31 MB 2025-02-14 08:37:21,456 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20290.83 MB 2025-02-14 08:37:21,627 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 08:37:21,627 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 08:37:21,627 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 08:37:21,627 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:37:21,627 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19135.62 MB 2025-02-14 08:37:21,627 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20963.27 MB 2025-02-14 08:37:21,627 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1827.65 MB 2025-02-14 08:37:21,627 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22630.37 MB 2025-02-14 08:37:21,628 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27634.17 MB 2025-02-14 08:37:21,628 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5003.80 MB 2025-02-14 08:37:21,628 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25483.95 MB 2025-02-14 08:37:21,628 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 08:37:21,628 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 08:37:21,628 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-14 08:37:21,628 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:37:21,628 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17594.48 MB 2025-02-14 08:37:21,628 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20963.27 MB 2025-02-14 08:37:21,628 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3368.79 MB 2025-02-14 08:37:21,628 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21091.06 MB 2025-02-14 08:37:21,628 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27634.17 MB 2025-02-14 08:37:21,628 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6543.11 MB 2025-02-14 08:37:21,628 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25483.95 MB 2025-02-14 08:37:21,770 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 08:37:21,770 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 08:37:21,770 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-14 08:37:21,770 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:37:21,770 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22213.11 MB 2025-02-14 08:37:21,770 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22838.22 MB 2025-02-14 08:37:21,770 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 625.11 MB 2025-02-14 08:37:21,770 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27634.17 MB 2025-02-14 08:37:21,770 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27971.81 MB 2025-02-14 08:37:21,770 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 337.64 MB 2025-02-14 08:37:21,770 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23415.07 MB 2025-02-14 08:37:21,789 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 08:37:21,789 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 08:37:21,789 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:37:21,789 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:37:21,789 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23174.72 MB 2025-02-14 08:37:21,789 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23393.62 MB 2025-02-14 08:37:21,789 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 218.90 MB 2025-02-14 08:37:21,789 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27971.81 MB 2025-02-14 08:37:21,789 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27971.81 MB 2025-02-14 08:37:21,789 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:37:21,789 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23514.19 MB 2025-02-14 08:37:21,790 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 08:37:21,790 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 08:37:21,790 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.63 seconds 2025-02-14 08:37:21,790 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:37:21,790 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14243.88 MB 2025-02-14 08:37:21,790 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23594.69 MB 2025-02-14 08:37:21,790 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 9350.81 MB 2025-02-14 08:37:21,790 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55866.03 MB 2025-02-14 08:37:21,791 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27971.81 MB 2025-02-14 08:37:21,791 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27894.22 MB 2025-02-14 08:37:21,791 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23594.69 MB 2025-02-14 08:37:22,058 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 08:37:22,058 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 08:37:22,058 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 08:37:22,058 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:37:22,058 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23594.69 MB 2025-02-14 08:37:22,058 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26608.73 MB 2025-02-14 08:37:22,058 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-14 08:37:22,058 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27971.81 MB 2025-02-14 08:37:22,058 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27971.81 MB 2025-02-14 08:37:22,058 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:37:22,058 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26910.09 MB 2025-02-14 08:37:22,076 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 08:37:22,077 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2,'] 2025-02-14 08:37:22,083 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 08:37:22,083 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 08:37:22,083 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:37:22,083 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:37:22,083 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18898.91 MB 2025-02-14 08:37:22,083 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27337.93 MB 2025-02-14 08:37:22,083 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 08:37:22,083 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27971.81 MB 2025-02-14 08:37:22,083 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38461.77 MB 2025-02-14 08:37:22,083 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 08:37:22,083 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27337.93 MB 2025-02-14 08:37:22,245 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 08:37:22,246 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:37:22,246 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 08:37:22,247 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:37:22,247 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 08:37:22,252 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 08:37:22,253 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:37:22,253 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 08:37:22,253 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2,'] 2025-02-14 08:38:02,193 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:38:02,193 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 08:38:02,198 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 08:38:02,201 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:38:02,201 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 468, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 08:38:02,202 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:38:02,202 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 468, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 08:38:09,481 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 08:38:09,481 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 08:38:09,481 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.27 seconds 2025-02-14 08:38:09,481 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:38:09,481 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16229.81 MB 2025-02-14 08:38:09,481 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17886.56 MB 2025-02-14 08:38:09,481 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1656.75 MB 2025-02-14 08:38:09,481 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51046.78 MB 2025-02-14 08:38:09,481 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21609.05 MB 2025-02-14 08:38:09,481 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29437.72 MB 2025-02-14 08:38:09,481 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26835.40 MB 2025-02-14 08:38:09,515 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 08:38:09,515 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 08:38:09,515 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 08:38:09,515 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:38:09,515 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17886.56 MB 2025-02-14 08:38:09,515 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18211.89 MB 2025-02-14 08:38:09,515 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 325.33 MB 2025-02-14 08:38:09,515 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21609.05 MB 2025-02-14 08:38:09,515 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28263.32 MB 2025-02-14 08:38:09,515 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6654.26 MB 2025-02-14 08:38:09,515 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25239.12 MB 2025-02-14 08:38:11,443 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 08:38:11,443 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 08:38:11,443 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 08:38:11,443 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:38:11,443 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18211.89 MB 2025-02-14 08:38:11,443 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18742.73 MB 2025-02-14 08:38:11,443 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 08:38:11,443 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28263.32 MB 2025-02-14 08:38:11,443 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21133.00 MB 2025-02-14 08:38:11,443 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7130.32 MB 2025-02-14 08:38:11,443 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22723.10 MB 2025-02-14 08:38:11,457 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 08:38:11,457 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 08:38:11,457 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:38:11,457 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:38:11,457 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18742.73 MB 2025-02-14 08:38:11,457 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20632.26 MB 2025-02-14 08:38:11,457 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 08:38:11,457 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21133.00 MB 2025-02-14 08:38:11,457 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24436.02 MB 2025-02-14 08:38:11,457 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 08:38:11,457 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22049.69 MB 2025-02-14 08:38:11,670 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 08:38:11,670 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 08:38:11,670 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 08:38:11,670 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:38:11,670 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20632.26 MB 2025-02-14 08:38:11,670 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22874.12 MB 2025-02-14 08:38:11,670 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 08:38:11,670 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24436.02 MB 2025-02-14 08:38:11,670 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30570.18 MB 2025-02-14 08:38:11,670 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 08:38:11,670 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28418.40 MB 2025-02-14 08:38:11,671 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 08:38:11,671 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 08:38:11,671 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 08:38:11,671 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:38:11,671 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18742.73 MB 2025-02-14 08:38:11,671 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22874.12 MB 2025-02-14 08:38:11,671 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 08:38:11,671 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21133.00 MB 2025-02-14 08:38:11,671 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30570.18 MB 2025-02-14 08:38:11,671 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9437.18 MB 2025-02-14 08:38:11,671 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28418.40 MB 2025-02-14 08:38:11,847 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 08:38:11,847 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 08:38:11,847 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 08:38:11,847 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:38:11,847 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24407.66 MB 2025-02-14 08:38:11,847 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25174.66 MB 2025-02-14 08:38:11,847 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 08:38:11,847 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30570.18 MB 2025-02-14 08:38:11,847 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30985.42 MB 2025-02-14 08:38:11,847 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 08:38:11,847 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25882.45 MB 2025-02-14 08:38:11,866 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 08:38:11,866 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 08:38:11,866 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:38:11,866 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:38:11,866 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25587.55 MB 2025-02-14 08:38:11,866 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25820.30 MB 2025-02-14 08:38:11,866 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 232.75 MB 2025-02-14 08:38:11,866 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30985.42 MB 2025-02-14 08:38:11,866 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30985.42 MB 2025-02-14 08:38:11,866 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:38:11,866 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26013.11 MB 2025-02-14 08:38:11,867 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 08:38:11,867 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 08:38:11,867 - resource_logging.py:150 - __exit__ - DEBUG - Time: 9.66 seconds 2025-02-14 08:38:11,867 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:38:11,867 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14599.26 MB 2025-02-14 08:38:11,867 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26021.37 MB 2025-02-14 08:38:11,867 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11422.11 MB 2025-02-14 08:38:11,867 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51046.78 MB 2025-02-14 08:38:11,867 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30985.42 MB 2025-02-14 08:38:11,867 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20061.36 MB 2025-02-14 08:38:11,867 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26021.37 MB 2025-02-14 08:38:12,136 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 08:38:12,136 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 08:38:12,136 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 08:38:12,136 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:38:12,136 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26021.37 MB 2025-02-14 08:38:12,136 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19603.64 MB 2025-02-14 08:38:12,136 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6417.72 MB 2025-02-14 08:38:12,136 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30985.42 MB 2025-02-14 08:38:12,136 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30985.42 MB 2025-02-14 08:38:12,136 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:38:12,136 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28533.04 MB 2025-02-14 08:38:12,154 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 08:38:12,154 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 08:38:12,160 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 08:38:12,160 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 08:38:12,160 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:38:12,160 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:38:12,160 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19603.64 MB 2025-02-14 08:38:12,160 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28042.67 MB 2025-02-14 08:38:12,160 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 08:38:12,160 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30985.42 MB 2025-02-14 08:38:12,160 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41475.38 MB 2025-02-14 08:38:12,160 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 08:38:12,160 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28042.67 MB 2025-02-14 08:38:12,328 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 08:38:12,330 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:38:12,330 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 08:38:12,331 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:38:12,331 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 08:38:12,336 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 08:38:12,337 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:38:12,337 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 08:38:12,337 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 08:38:36,786 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:38:36,786 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 08:38:36,791 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 08:38:36,794 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:38:36,794 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 761, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 08:38:36,795 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:38:36,795 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 761, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 08:38:48,553 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 08:38:48,553 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 08:38:48,553 - resource_logging.py:150 - __exit__ - DEBUG - Time: 11.75 seconds 2025-02-14 08:38:48,553 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:38:48,553 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18271.48 MB 2025-02-14 08:38:48,553 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20964.61 MB 2025-02-14 08:38:48,553 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2693.14 MB 2025-02-14 08:38:48,553 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54060.38 MB 2025-02-14 08:38:48,553 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24618.47 MB 2025-02-14 08:38:48,553 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29441.92 MB 2025-02-14 08:38:48,553 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29781.28 MB 2025-02-14 08:38:48,602 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 08:38:48,602 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 08:38:48,602 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-14 08:38:48,602 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:38:48,602 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20964.61 MB 2025-02-14 08:38:48,602 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19734.05 MB 2025-02-14 08:38:48,602 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1230.56 MB 2025-02-14 08:38:48,602 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24618.47 MB 2025-02-14 08:38:48,602 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32694.60 MB 2025-02-14 08:38:48,602 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8076.13 MB 2025-02-14 08:38:48,602 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29569.42 MB 2025-02-14 08:38:50,518 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 08:38:50,519 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 08:38:50,519 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 08:38:50,519 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:38:50,519 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19734.05 MB 2025-02-14 08:38:50,519 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20264.89 MB 2025-02-14 08:38:50,519 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 08:38:50,519 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32694.60 MB 2025-02-14 08:38:50,519 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26034.04 MB 2025-02-14 08:38:50,519 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6660.55 MB 2025-02-14 08:38:50,519 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24244.23 MB 2025-02-14 08:38:50,533 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 08:38:50,533 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 08:38:50,533 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:38:50,533 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:38:50,533 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20264.89 MB 2025-02-14 08:38:50,533 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22154.43 MB 2025-02-14 08:38:50,533 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 08:38:50,533 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26034.04 MB 2025-02-14 08:38:50,533 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26034.04 MB 2025-02-14 08:38:50,533 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:38:50,533 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23571.86 MB 2025-02-14 08:38:50,743 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 08:38:50,743 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 08:38:50,743 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 08:38:50,743 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:38:50,743 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22154.43 MB 2025-02-14 08:38:50,743 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24396.28 MB 2025-02-14 08:38:50,743 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 08:38:50,743 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26034.04 MB 2025-02-14 08:38:50,743 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32168.21 MB 2025-02-14 08:38:50,743 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 08:38:50,743 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29940.57 MB 2025-02-14 08:38:50,744 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 08:38:50,744 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 08:38:50,744 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 08:38:50,744 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:38:50,744 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20264.89 MB 2025-02-14 08:38:50,744 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24396.28 MB 2025-02-14 08:38:50,744 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 08:38:50,744 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26034.04 MB 2025-02-14 08:38:50,744 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32168.21 MB 2025-02-14 08:38:50,744 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 08:38:50,744 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29940.57 MB 2025-02-14 08:38:50,911 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 08:38:50,911 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 08:38:50,911 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 08:38:50,911 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:38:50,911 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25929.83 MB 2025-02-14 08:38:50,911 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26696.83 MB 2025-02-14 08:38:50,911 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 08:38:50,911 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32168.21 MB 2025-02-14 08:38:50,911 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32583.45 MB 2025-02-14 08:38:50,911 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 08:38:50,911 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27404.62 MB 2025-02-14 08:38:50,930 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 08:38:50,930 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 08:38:50,930 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:38:50,930 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:38:50,930 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27109.72 MB 2025-02-14 08:38:50,930 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27338.26 MB 2025-02-14 08:38:50,930 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.54 MB 2025-02-14 08:38:50,930 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32583.45 MB 2025-02-14 08:38:50,930 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32583.45 MB 2025-02-14 08:38:50,930 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:38:50,930 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27511.50 MB 2025-02-14 08:38:50,931 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 08:38:50,931 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 08:38:50,931 - resource_logging.py:150 - __exit__ - DEBUG - Time: 14.13 seconds 2025-02-14 08:38:50,931 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:38:50,931 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15620.09 MB 2025-02-14 08:38:50,931 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27538.72 MB 2025-02-14 08:38:50,931 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11918.63 MB 2025-02-14 08:38:50,931 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54060.38 MB 2025-02-14 08:38:50,931 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32583.45 MB 2025-02-14 08:38:50,931 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21476.93 MB 2025-02-14 08:38:50,931 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27538.72 MB 2025-02-14 08:38:51,198 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 08:38:51,198 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 08:38:51,198 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 08:38:51,199 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:38:51,199 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27538.72 MB 2025-02-14 08:38:51,199 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20614.96 MB 2025-02-14 08:38:51,199 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6923.76 MB 2025-02-14 08:38:51,199 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32583.45 MB 2025-02-14 08:38:51,199 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32583.45 MB 2025-02-14 08:38:51,199 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:38:51,199 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30042.71 MB 2025-02-14 08:38:51,216 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8137, cut from 8139 2025-02-14 08:38:51,216 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 08:38:51,222 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 08:38:51,222 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 08:38:51,222 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:38:51,222 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:38:51,222 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20614.96 MB 2025-02-14 08:38:51,222 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29027.91 MB 2025-02-14 08:38:51,222 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8412.95 MB 2025-02-14 08:38:51,222 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32583.45 MB 2025-02-14 08:38:51,222 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36765.17 MB 2025-02-14 08:38:51,222 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4181.72 MB 2025-02-14 08:38:51,222 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29027.91 MB 2025-02-14 08:38:51,384 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7929] 2025-02-14 08:38:51,386 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:38:51,386 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 08:38:51,387 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:38:51,387 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 08:38:51,391 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 08:38:51,392 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:38:51,392 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 08:38:51,393 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 08:39:49,940 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:39:49,941 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 08:39:49,945 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 08:39:49,949 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:39:49,949 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 535, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 08:39:49,950 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:39:49,950 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 535, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 08:39:58,196 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 08:39:58,196 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 08:39:58,196 - resource_logging.py:150 - __exit__ - DEBUG - Time: 8.24 seconds 2025-02-14 08:39:58,196 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:39:58,196 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16696.67 MB 2025-02-14 08:39:58,196 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18590.40 MB 2025-02-14 08:39:58,196 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1893.73 MB 2025-02-14 08:39:58,196 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45128.61 MB 2025-02-14 08:39:58,196 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22106.08 MB 2025-02-14 08:39:58,196 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23022.53 MB 2025-02-14 08:39:58,196 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27527.80 MB 2025-02-14 08:39:58,232 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 08:39:58,232 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 08:39:58,232 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 08:39:58,232 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:39:58,232 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18590.40 MB 2025-02-14 08:39:58,232 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18560.20 MB 2025-02-14 08:39:58,232 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -30.20 MB 2025-02-14 08:39:58,232 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22106.08 MB 2025-02-14 08:39:58,232 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28993.13 MB 2025-02-14 08:39:58,232 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6887.05 MB 2025-02-14 08:39:58,232 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26184.75 MB 2025-02-14 08:40:00,160 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 08:40:00,160 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 08:40:00,160 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 08:40:00,160 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:40:00,160 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18560.20 MB 2025-02-14 08:40:00,160 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19091.04 MB 2025-02-14 08:40:00,160 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 08:40:00,160 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28993.13 MB 2025-02-14 08:40:00,160 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22336.77 MB 2025-02-14 08:40:00,160 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6656.36 MB 2025-02-14 08:40:00,160 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23071.41 MB 2025-02-14 08:40:00,173 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 08:40:00,173 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 08:40:00,173 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:40:00,173 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:40:00,173 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19091.04 MB 2025-02-14 08:40:00,173 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20980.57 MB 2025-02-14 08:40:00,173 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 08:40:00,173 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22336.77 MB 2025-02-14 08:40:00,173 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25167.92 MB 2025-02-14 08:40:00,174 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-14 08:40:00,174 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22398.00 MB 2025-02-14 08:40:00,385 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 08:40:00,385 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 08:40:00,385 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 08:40:00,385 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:40:00,385 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20980.57 MB 2025-02-14 08:40:00,385 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23222.43 MB 2025-02-14 08:40:00,385 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 08:40:00,385 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25167.92 MB 2025-02-14 08:40:00,385 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31302.09 MB 2025-02-14 08:40:00,385 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 08:40:00,385 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28766.71 MB 2025-02-14 08:40:00,386 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 08:40:00,386 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 08:40:00,386 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 08:40:00,386 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:40:00,386 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19091.04 MB 2025-02-14 08:40:00,386 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23222.43 MB 2025-02-14 08:40:00,386 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 08:40:00,386 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22336.77 MB 2025-02-14 08:40:00,386 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31302.09 MB 2025-02-14 08:40:00,386 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8965.32 MB 2025-02-14 08:40:00,386 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28766.71 MB 2025-02-14 08:40:00,556 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 08:40:00,556 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 08:40:00,556 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 08:40:00,556 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:40:00,556 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24755.97 MB 2025-02-14 08:40:00,556 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25522.97 MB 2025-02-14 08:40:00,556 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 08:40:00,556 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31302.09 MB 2025-02-14 08:40:00,556 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31717.33 MB 2025-02-14 08:40:00,556 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 08:40:00,556 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26230.76 MB 2025-02-14 08:40:00,575 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 08:40:00,575 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 08:40:00,575 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:40:00,575 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:40:00,575 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25935.86 MB 2025-02-14 08:40:00,575 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26165.68 MB 2025-02-14 08:40:00,575 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.82 MB 2025-02-14 08:40:00,575 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31717.33 MB 2025-02-14 08:40:00,575 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31717.33 MB 2025-02-14 08:40:00,575 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:40:00,575 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26333.39 MB 2025-02-14 08:40:00,576 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 08:40:00,576 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 08:40:00,576 - resource_logging.py:150 - __exit__ - DEBUG - Time: 10.62 seconds 2025-02-14 08:40:00,576 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:40:00,576 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14832.69 MB 2025-02-14 08:40:00,576 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26366.76 MB 2025-02-14 08:40:00,576 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11534.07 MB 2025-02-14 08:40:00,576 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45128.61 MB 2025-02-14 08:40:00,576 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31717.33 MB 2025-02-14 08:40:00,576 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13411.29 MB 2025-02-14 08:40:00,576 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26366.76 MB 2025-02-14 08:40:00,844 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 08:40:00,844 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 08:40:00,844 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 08:40:00,844 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:40:00,844 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26366.76 MB 2025-02-14 08:40:00,844 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19837.08 MB 2025-02-14 08:40:00,844 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6529.68 MB 2025-02-14 08:40:00,844 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31717.33 MB 2025-02-14 08:40:00,844 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31717.33 MB 2025-02-14 08:40:00,844 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:40:00,844 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28878.42 MB 2025-02-14 08:40:00,862 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 08:40:00,862 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-14 08:40:00,868 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 08:40:00,868 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 08:40:00,868 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:40:00,869 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:40:00,869 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19837.08 MB 2025-02-14 08:40:00,869 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28276.10 MB 2025-02-14 08:40:00,869 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 08:40:00,869 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31717.33 MB 2025-02-14 08:40:00,869 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42207.28 MB 2025-02-14 08:40:00,869 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 08:40:00,869 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28276.10 MB 2025-02-14 08:40:01,038 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 08:40:01,040 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:40:01,040 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 08:40:01,041 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:40:01,041 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 08:40:01,046 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 08:40:01,047 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:40:01,047 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 08:40:01,047 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-14 08:40:26,055 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:40:26,055 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 08:40:26,060 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 08:40:26,063 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:40:26,063 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1504, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 08:40:26,064 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:40:26,064 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1504, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 08:40:49,201 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 08:40:49,201 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 08:40:49,201 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.13 seconds 2025-02-14 08:40:49,201 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:40:49,201 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23448.82 MB 2025-02-14 08:40:49,201 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28771.39 MB 2025-02-14 08:40:49,201 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5322.57 MB 2025-02-14 08:40:49,201 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54792.29 MB 2025-02-14 08:40:49,201 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39013.32 MB 2025-02-14 08:40:49,201 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15778.97 MB 2025-02-14 08:40:49,201 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37677.34 MB 2025-02-14 08:40:49,287 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 08:40:49,287 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 08:40:49,287 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 08:40:49,287 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:40:49,287 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28771.39 MB 2025-02-14 08:40:49,287 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23596.68 MB 2025-02-14 08:40:49,287 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5174.71 MB 2025-02-14 08:40:49,287 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39013.32 MB 2025-02-14 08:40:49,287 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49991.91 MB 2025-02-14 08:40:49,287 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10978.59 MB 2025-02-14 08:40:49,287 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44556.03 MB 2025-02-14 08:40:51,195 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 08:40:51,196 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 08:40:51,196 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 08:40:51,196 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:40:51,196 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23596.68 MB 2025-02-14 08:40:51,196 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24127.52 MB 2025-02-14 08:40:51,196 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 08:40:51,196 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49991.91 MB 2025-02-14 08:40:51,196 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33690.75 MB 2025-02-14 08:40:51,196 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16301.16 MB 2025-02-14 08:40:51,196 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28106.85 MB 2025-02-14 08:40:51,209 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 08:40:51,209 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 08:40:51,209 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:40:51,209 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:40:51,209 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24127.52 MB 2025-02-14 08:40:51,209 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26017.05 MB 2025-02-14 08:40:51,209 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 08:40:51,209 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33690.75 MB 2025-02-14 08:40:51,209 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33690.75 MB 2025-02-14 08:40:51,209 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:40:51,209 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27434.48 MB 2025-02-14 08:40:51,423 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 08:40:51,423 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 08:40:51,423 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 08:40:51,423 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:40:51,423 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26017.05 MB 2025-02-14 08:40:51,423 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28258.91 MB 2025-02-14 08:40:51,423 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 08:40:51,423 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33690.75 MB 2025-02-14 08:40:51,423 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37465.62 MB 2025-02-14 08:40:51,423 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 08:40:51,423 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33803.19 MB 2025-02-14 08:40:51,424 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 08:40:51,424 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 08:40:51,424 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 08:40:51,424 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:40:51,424 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24127.52 MB 2025-02-14 08:40:51,424 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28258.91 MB 2025-02-14 08:40:51,424 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 08:40:51,424 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33690.75 MB 2025-02-14 08:40:51,424 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37465.62 MB 2025-02-14 08:40:51,424 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 08:40:51,424 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33803.19 MB 2025-02-14 08:40:51,597 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 08:40:51,597 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 08:40:51,597 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 08:40:51,597 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:40:51,597 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29792.45 MB 2025-02-14 08:40:51,597 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30559.45 MB 2025-02-14 08:40:51,597 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 08:40:51,597 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37465.62 MB 2025-02-14 08:40:51,597 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37878.76 MB 2025-02-14 08:40:51,597 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 08:40:51,597 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31267.24 MB 2025-02-14 08:40:51,616 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 08:40:51,616 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 08:40:51,616 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:40:51,616 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:40:51,616 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30972.34 MB 2025-02-14 08:40:51,616 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31201.07 MB 2025-02-14 08:40:51,616 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.73 MB 2025-02-14 08:40:51,616 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37878.76 MB 2025-02-14 08:40:51,616 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37878.76 MB 2025-02-14 08:40:51,616 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:40:51,616 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31423.49 MB 2025-02-14 08:40:51,617 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 08:40:51,617 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 08:40:51,617 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.55 seconds 2025-02-14 08:40:51,617 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:40:51,617 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18208.76 MB 2025-02-14 08:40:51,617 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31401.80 MB 2025-02-14 08:40:51,617 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13193.04 MB 2025-02-14 08:40:51,617 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54792.29 MB 2025-02-14 08:40:51,617 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37878.76 MB 2025-02-14 08:40:51,617 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16913.53 MB 2025-02-14 08:40:51,617 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31423.49 MB 2025-02-14 08:40:51,885 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 08:40:51,886 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 08:40:51,886 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 08:40:51,886 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:40:51,886 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31401.80 MB 2025-02-14 08:40:51,886 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23208.51 MB 2025-02-14 08:40:51,886 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8193.29 MB 2025-02-14 08:40:51,886 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37878.76 MB 2025-02-14 08:40:51,886 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37878.76 MB 2025-02-14 08:40:51,886 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:40:51,886 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33909.16 MB 2025-02-14 08:40:51,903 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8148, cut from 8150 2025-02-14 08:40:51,903 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-14 08:40:51,909 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 08:40:51,910 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 08:40:51,910 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:40:51,910 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:40:51,910 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23208.51 MB 2025-02-14 08:40:51,910 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31632.75 MB 2025-02-14 08:40:51,910 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8424.24 MB 2025-02-14 08:40:51,910 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37878.76 MB 2025-02-14 08:40:51,910 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42066.77 MB 2025-02-14 08:40:51,910 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4188.01 MB 2025-02-14 08:40:51,910 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31632.75 MB 2025-02-14 08:40:52,078 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7940] 2025-02-14 08:40:52,079 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:40:52,080 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 08:40:52,080 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:40:52,081 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 08:40:52,085 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 08:40:52,087 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:40:52,087 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 08:40:52,087 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-14 08:41:03,638 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:41:03,638 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 08:41:03,643 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 08:41:03,647 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:41:03,647 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 566, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 08:41:03,648 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:41:03,648 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 566, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 08:41:12,440 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 08:41:12,440 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 08:41:12,440 - resource_logging.py:150 - __exit__ - DEBUG - Time: 8.79 seconds 2025-02-14 08:41:12,440 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:41:12,440 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16912.69 MB 2025-02-14 08:41:12,440 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18915.73 MB 2025-02-14 08:41:12,440 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2003.04 MB 2025-02-14 08:41:12,440 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50442.80 MB 2025-02-14 08:41:12,440 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23110.62 MB 2025-02-14 08:41:12,440 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27332.18 MB 2025-02-14 08:41:12,440 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27743.82 MB 2025-02-14 08:41:12,477 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 08:41:12,478 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 08:41:12,478 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-14 08:41:12,478 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:41:12,478 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18915.73 MB 2025-02-14 08:41:12,478 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18720.31 MB 2025-02-14 08:41:12,478 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -195.42 MB 2025-02-14 08:41:12,478 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23110.62 MB 2025-02-14 08:41:12,478 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29920.07 MB 2025-02-14 08:41:12,478 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6809.45 MB 2025-02-14 08:41:12,478 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27138.21 MB 2025-02-14 08:41:14,426 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 08:41:14,426 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 08:41:14,426 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.95 seconds 2025-02-14 08:41:14,426 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:41:14,426 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18720.31 MB 2025-02-14 08:41:14,426 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19251.15 MB 2025-02-14 08:41:14,426 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 08:41:14,426 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29920.07 MB 2025-02-14 08:41:14,426 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22521.32 MB 2025-02-14 08:41:14,426 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7398.75 MB 2025-02-14 08:41:14,426 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23231.52 MB 2025-02-14 08:41:14,439 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 08:41:14,440 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 08:41:14,440 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:41:14,440 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:41:14,440 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19251.15 MB 2025-02-14 08:41:14,440 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21140.68 MB 2025-02-14 08:41:14,440 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 08:41:14,440 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22521.32 MB 2025-02-14 08:41:14,440 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25352.47 MB 2025-02-14 08:41:14,440 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-14 08:41:14,440 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22558.11 MB 2025-02-14 08:41:14,648 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 08:41:14,648 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 08:41:14,648 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 08:41:14,648 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:41:14,648 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21140.68 MB 2025-02-14 08:41:14,648 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23382.54 MB 2025-02-14 08:41:14,648 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 08:41:14,648 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25352.47 MB 2025-02-14 08:41:14,648 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31014.78 MB 2025-02-14 08:41:14,648 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 08:41:14,648 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28926.82 MB 2025-02-14 08:41:14,649 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 08:41:14,649 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 08:41:14,649 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 08:41:14,649 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:41:14,649 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19251.15 MB 2025-02-14 08:41:14,649 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23382.54 MB 2025-02-14 08:41:14,649 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 08:41:14,649 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22521.32 MB 2025-02-14 08:41:14,649 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31014.78 MB 2025-02-14 08:41:14,649 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8493.47 MB 2025-02-14 08:41:14,649 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28926.82 MB 2025-02-14 08:41:14,815 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 08:41:14,815 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 08:41:14,815 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 08:41:14,815 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:41:14,815 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24916.08 MB 2025-02-14 08:41:14,815 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25683.08 MB 2025-02-14 08:41:14,815 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 08:41:14,815 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31014.78 MB 2025-02-14 08:41:14,815 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31427.92 MB 2025-02-14 08:41:14,815 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 08:41:14,815 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26390.87 MB 2025-02-14 08:41:14,834 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 08:41:14,834 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 08:41:14,834 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:41:14,834 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:41:14,834 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26095.97 MB 2025-02-14 08:41:14,834 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26324.53 MB 2025-02-14 08:41:14,834 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.56 MB 2025-02-14 08:41:14,834 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31427.92 MB 2025-02-14 08:41:14,834 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31427.92 MB 2025-02-14 08:41:14,834 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:41:14,834 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26507.90 MB 2025-02-14 08:41:14,835 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 08:41:14,835 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 08:41:14,835 - resource_logging.py:150 - __exit__ - DEBUG - Time: 11.19 seconds 2025-02-14 08:41:14,835 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:41:14,835 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14940.70 MB 2025-02-14 08:41:14,835 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26525.21 MB 2025-02-14 08:41:14,835 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11584.51 MB 2025-02-14 08:41:14,835 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50442.80 MB 2025-02-14 08:41:14,835 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31427.92 MB 2025-02-14 08:41:14,835 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19014.88 MB 2025-02-14 08:41:14,835 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26525.21 MB 2025-02-14 08:41:15,103 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 08:41:15,103 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 08:41:15,103 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 08:41:15,103 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:41:15,103 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26525.21 MB 2025-02-14 08:41:15,103 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19938.99 MB 2025-02-14 08:41:15,103 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6586.22 MB 2025-02-14 08:41:15,103 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31427.92 MB 2025-02-14 08:41:15,103 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31427.92 MB 2025-02-14 08:41:15,103 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:41:15,103 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29031.96 MB 2025-02-14 08:41:15,121 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8146, cut from 8148 2025-02-14 08:41:15,121 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 08:41:15,127 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 08:41:15,128 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 08:41:15,128 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:41:15,128 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:41:15,128 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19938.99 MB 2025-02-14 08:41:15,128 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28361.32 MB 2025-02-14 08:41:15,128 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8422.33 MB 2025-02-14 08:41:15,128 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31427.92 MB 2025-02-14 08:41:15,128 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41896.90 MB 2025-02-14 08:41:15,128 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10468.98 MB 2025-02-14 08:41:15,128 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28361.32 MB 2025-02-14 08:41:15,292 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7938] 2025-02-14 08:41:15,294 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:41:15,294 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 08:41:15,295 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:41:15,295 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 08:41:15,300 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 08:41:15,301 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:41:15,301 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 08:41:15,301 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 08:41:34,722 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:41:34,722 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 08:41:34,727 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 08:41:34,730 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:41:34,730 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 224, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 08:41:34,731 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:41:34,731 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 224, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 08:41:38,237 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 08:41:38,237 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 08:41:38,237 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.50 seconds 2025-02-14 08:41:38,237 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:41:38,237 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14529.57 MB 2025-02-14 08:41:38,237 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15322.30 MB 2025-02-14 08:41:38,237 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 792.72 MB 2025-02-14 08:41:38,237 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54456.75 MB 2025-02-14 08:41:38,237 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18324.91 MB 2025-02-14 08:41:38,237 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -36131.83 MB 2025-02-14 08:41:38,237 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24228.24 MB 2025-02-14 08:41:38,252 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 08:41:38,252 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 08:41:38,252 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:41:38,252 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:41:38,252 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15322.30 MB 2025-02-14 08:41:38,252 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15376.94 MB 2025-02-14 08:41:38,252 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 54.65 MB 2025-02-14 08:41:38,252 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18324.91 MB 2025-02-14 08:41:38,252 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18951.96 MB 2025-02-14 08:41:38,252 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 627.05 MB 2025-02-14 08:41:38,253 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17866.58 MB 2025-02-14 08:41:39,105 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 08:41:39,105 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 08:41:39,105 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.85 seconds 2025-02-14 08:41:39,105 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:41:39,105 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15376.94 MB 2025-02-14 08:41:39,105 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15611.84 MB 2025-02-14 08:41:39,105 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 234.90 MB 2025-02-14 08:41:39,105 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18951.96 MB 2025-02-14 08:41:39,105 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19354.62 MB 2025-02-14 08:41:39,105 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 402.65 MB 2025-02-14 08:41:39,105 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19548.42 MB 2025-02-14 08:41:39,113 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 08:41:39,113 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 08:41:39,113 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:41:39,113 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:41:39,113 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15611.77 MB 2025-02-14 08:41:39,113 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16447.69 MB 2025-02-14 08:41:39,113 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 835.92 MB 2025-02-14 08:41:39,113 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19354.62 MB 2025-02-14 08:41:39,113 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19354.62 MB 2025-02-14 08:41:39,113 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:41:39,114 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17074.91 MB 2025-02-14 08:41:39,211 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 08:41:39,211 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 08:41:39,211 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 08:41:39,211 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:41:39,211 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16447.69 MB 2025-02-14 08:41:39,211 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17439.75 MB 2025-02-14 08:41:39,211 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 992.06 MB 2025-02-14 08:41:39,211 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19354.62 MB 2025-02-14 08:41:39,211 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21451.77 MB 2025-02-14 08:41:39,211 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2097.15 MB 2025-02-14 08:41:39,211 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19893.06 MB 2025-02-14 08:41:39,212 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 08:41:39,212 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 08:41:39,212 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 08:41:39,212 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:41:39,212 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15611.77 MB 2025-02-14 08:41:39,212 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17439.75 MB 2025-02-14 08:41:39,212 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1827.97 MB 2025-02-14 08:41:39,212 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19354.62 MB 2025-02-14 08:41:39,212 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21451.77 MB 2025-02-14 08:41:39,212 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2097.15 MB 2025-02-14 08:41:39,212 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19893.06 MB 2025-02-14 08:41:39,287 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 08:41:39,287 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 08:41:39,287 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 08:41:39,287 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:41:39,287 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18118.34 MB 2025-02-14 08:41:39,287 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18457.74 MB 2025-02-14 08:41:39,287 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 339.40 MB 2025-02-14 08:41:39,287 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21451.77 MB 2025-02-14 08:41:39,287 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21632.12 MB 2025-02-14 08:41:39,287 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 180.36 MB 2025-02-14 08:41:39,287 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18777.17 MB 2025-02-14 08:41:39,297 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 08:41:39,297 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 08:41:39,297 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:41:39,297 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:41:39,297 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18640.45 MB 2025-02-14 08:41:39,297 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18869.89 MB 2025-02-14 08:41:39,297 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.44 MB 2025-02-14 08:41:39,297 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21632.12 MB 2025-02-14 08:41:39,297 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21632.12 MB 2025-02-14 08:41:39,297 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:41:39,297 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18893.06 MB 2025-02-14 08:41:39,298 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 08:41:39,298 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 08:41:39,298 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.57 seconds 2025-02-14 08:41:39,298 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:41:39,298 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13749.14 MB 2025-02-14 08:41:39,298 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19070.96 MB 2025-02-14 08:41:39,298 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5321.82 MB 2025-02-14 08:41:39,298 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54456.75 MB 2025-02-14 08:41:39,298 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21632.12 MB 2025-02-14 08:41:39,298 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32824.62 MB 2025-02-14 08:41:39,298 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19070.96 MB 2025-02-14 08:41:39,565 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 08:41:39,565 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 08:41:39,565 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 08:41:39,565 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:41:39,565 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19070.96 MB 2025-02-14 08:41:39,565 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17701.91 MB 2025-02-14 08:41:39,565 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1369.05 MB 2025-02-14 08:41:39,565 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21632.12 MB 2025-02-14 08:41:39,565 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21632.12 MB 2025-02-14 08:41:39,565 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:41:39,565 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19305.86 MB 2025-02-14 08:41:39,583 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 08:41:39,583 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 08:41:39,589 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 08:41:39,589 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 08:41:39,589 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:41:39,589 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:41:39,589 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17701.91 MB 2025-02-14 08:41:39,589 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26140.94 MB 2025-02-14 08:41:39,589 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 08:41:39,589 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21632.12 MB 2025-02-14 08:41:39,589 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32122.08 MB 2025-02-14 08:41:39,589 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 08:41:39,589 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26140.94 MB 2025-02-14 08:41:39,753 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 08:41:39,754 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:41:39,754 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 08:41:39,755 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:41:39,755 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 08:41:39,760 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 08:41:39,761 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:41:39,761 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 08:41:39,761 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 08:41:51,033 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:41:51,033 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 08:41:51,038 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 08:41:51,041 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:41:51,041 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 464, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 08:41:51,042 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:41:51,042 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 464, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 08:41:58,245 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 08:41:58,245 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 08:41:58,245 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.20 seconds 2025-02-14 08:41:58,245 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:41:58,245 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16201.93 MB 2025-02-14 08:41:58,245 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17844.00 MB 2025-02-14 08:41:58,245 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1642.07 MB 2025-02-14 08:41:58,245 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44707.09 MB 2025-02-14 08:41:58,245 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21596.47 MB 2025-02-14 08:41:58,245 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23110.62 MB 2025-02-14 08:41:58,245 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26806.57 MB 2025-02-14 08:41:58,279 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 08:41:58,279 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 08:41:58,279 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 08:41:58,279 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:41:58,279 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17844.00 MB 2025-02-14 08:41:58,279 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18191.09 MB 2025-02-14 08:41:58,279 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 347.09 MB 2025-02-14 08:41:58,279 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21596.47 MB 2025-02-14 08:41:58,279 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28397.54 MB 2025-02-14 08:41:58,279 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6801.06 MB 2025-02-14 08:41:58,279 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25386.98 MB 2025-02-14 08:42:00,240 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 08:42:00,240 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 08:42:00,240 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.96 seconds 2025-02-14 08:42:00,240 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:42:00,240 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18191.09 MB 2025-02-14 08:42:00,240 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18721.93 MB 2025-02-14 08:42:00,240 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 08:42:00,240 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28397.54 MB 2025-02-14 08:42:00,240 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21135.10 MB 2025-02-14 08:42:00,240 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7262.44 MB 2025-02-14 08:42:00,240 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22702.30 MB 2025-02-14 08:42:00,254 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 08:42:00,254 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 08:42:00,254 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:42:00,254 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:42:00,254 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18721.93 MB 2025-02-14 08:42:00,254 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20611.47 MB 2025-02-14 08:42:00,254 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 08:42:00,254 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21135.10 MB 2025-02-14 08:42:00,254 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23966.25 MB 2025-02-14 08:42:00,254 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-14 08:42:00,254 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22028.90 MB 2025-02-14 08:42:00,465 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 08:42:00,466 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 08:42:00,466 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 08:42:00,466 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:42:00,466 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20611.47 MB 2025-02-14 08:42:00,466 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22853.32 MB 2025-02-14 08:42:00,466 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 08:42:00,466 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23966.25 MB 2025-02-14 08:42:00,466 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30572.28 MB 2025-02-14 08:42:00,466 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 08:42:00,466 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28397.60 MB 2025-02-14 08:42:00,466 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 08:42:00,466 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 08:42:00,466 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 08:42:00,466 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:42:00,466 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18721.93 MB 2025-02-14 08:42:00,466 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22853.32 MB 2025-02-14 08:42:00,466 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 08:42:00,466 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21135.10 MB 2025-02-14 08:42:00,466 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30572.28 MB 2025-02-14 08:42:00,466 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9437.18 MB 2025-02-14 08:42:00,466 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28397.60 MB 2025-02-14 08:42:00,632 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 08:42:00,632 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 08:42:00,632 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 08:42:00,632 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:42:00,632 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24386.86 MB 2025-02-14 08:42:00,632 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25153.87 MB 2025-02-14 08:42:00,632 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 08:42:00,632 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30572.28 MB 2025-02-14 08:42:00,632 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30985.42 MB 2025-02-14 08:42:00,632 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 08:42:00,632 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25861.66 MB 2025-02-14 08:42:00,651 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 08:42:00,651 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 08:42:00,651 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:42:00,651 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:42:00,651 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25566.76 MB 2025-02-14 08:42:00,651 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25795.89 MB 2025-02-14 08:42:00,651 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.13 MB 2025-02-14 08:42:00,651 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30985.42 MB 2025-02-14 08:42:00,651 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30985.42 MB 2025-02-14 08:42:00,651 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:42:00,651 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25965.29 MB 2025-02-14 08:42:00,652 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 08:42:00,652 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 08:42:00,652 - resource_logging.py:150 - __exit__ - DEBUG - Time: 9.61 seconds 2025-02-14 08:42:00,652 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:42:00,652 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14585.32 MB 2025-02-14 08:42:00,652 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25996.96 MB 2025-02-14 08:42:00,652 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11411.64 MB 2025-02-14 08:42:00,652 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44707.09 MB 2025-02-14 08:42:00,652 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30985.42 MB 2025-02-14 08:42:00,652 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13721.67 MB 2025-02-14 08:42:00,652 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25996.96 MB 2025-02-14 08:42:00,921 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 08:42:00,921 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 08:42:00,921 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 08:42:00,921 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:42:00,921 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25996.96 MB 2025-02-14 08:42:00,921 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19589.71 MB 2025-02-14 08:42:00,921 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6407.25 MB 2025-02-14 08:42:00,921 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30985.42 MB 2025-02-14 08:42:00,921 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30985.42 MB 2025-02-14 08:42:00,921 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:42:00,921 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28508.63 MB 2025-02-14 08:42:00,939 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 08:42:00,940 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 08:42:00,946 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 08:42:00,946 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 08:42:00,946 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:42:00,946 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:42:00,946 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19589.71 MB 2025-02-14 08:42:00,946 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28028.73 MB 2025-02-14 08:42:00,946 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 08:42:00,946 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30985.42 MB 2025-02-14 08:42:00,946 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41475.38 MB 2025-02-14 08:42:00,946 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 08:42:00,946 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28028.73 MB 2025-02-14 08:42:01,108 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 08:42:01,109 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:42:01,109 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 08:42:01,110 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:42:01,110 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 08:42:01,115 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 08:42:01,116 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:42:01,116 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 08:42:01,116 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 08:43:55,053 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:43:55,053 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 08:43:55,060 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 08:43:55,066 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:43:55,066 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 223, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 08:43:55,068 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:43:55,068 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 223, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 08:43:58,566 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 08:43:58,566 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 08:43:58,566 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.49 seconds 2025-02-14 08:43:58,566 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:43:58,566 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14522.61 MB 2025-02-14 08:43:58,566 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15311.79 MB 2025-02-14 08:43:58,566 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 789.18 MB 2025-02-14 08:43:58,566 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54060.38 MB 2025-02-14 08:43:58,566 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17853.05 MB 2025-02-14 08:43:58,566 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -36207.33 MB 2025-02-14 08:43:58,566 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24221.28 MB 2025-02-14 08:43:58,586 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 08:43:58,586 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 08:43:58,586 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:43:58,586 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:43:58,586 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15311.79 MB 2025-02-14 08:43:58,586 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15596.42 MB 2025-02-14 08:43:58,586 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 284.63 MB 2025-02-14 08:43:58,586 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17853.05 MB 2025-02-14 08:43:58,586 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19333.64 MB 2025-02-14 08:43:58,586 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1480.59 MB 2025-02-14 08:43:58,586 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18301.81 MB 2025-02-14 08:43:59,628 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 08:43:59,628 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 08:43:59,628 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.04 seconds 2025-02-14 08:43:59,628 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:43:59,628 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15596.42 MB 2025-02-14 08:43:59,628 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15873.78 MB 2025-02-14 08:43:59,628 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 277.36 MB 2025-02-14 08:43:59,628 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19333.64 MB 2025-02-14 08:43:59,628 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18524.14 MB 2025-02-14 08:43:59,628 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -809.50 MB 2025-02-14 08:43:59,628 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19852.83 MB 2025-02-14 08:43:59,639 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 08:43:59,639 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 08:43:59,640 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:43:59,640 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:43:59,640 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15873.78 MB 2025-02-14 08:43:59,640 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16860.82 MB 2025-02-14 08:43:59,640 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 987.04 MB 2025-02-14 08:43:59,640 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18524.14 MB 2025-02-14 08:43:59,640 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19019.07 MB 2025-02-14 08:43:59,640 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 494.93 MB 2025-02-14 08:43:59,640 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17601.43 MB 2025-02-14 08:43:59,780 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 08:43:59,780 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 08:43:59,780 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-14 08:43:59,780 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:43:59,780 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16860.82 MB 2025-02-14 08:43:59,780 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18032.45 MB 2025-02-14 08:43:59,780 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1171.63 MB 2025-02-14 08:43:59,780 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19019.07 MB 2025-02-14 08:43:59,780 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21988.64 MB 2025-02-14 08:43:59,780 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2969.57 MB 2025-02-14 08:43:59,780 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20929.31 MB 2025-02-14 08:43:59,781 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 08:43:59,781 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 08:43:59,781 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 08:43:59,781 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:43:59,781 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15873.78 MB 2025-02-14 08:43:59,781 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18032.45 MB 2025-02-14 08:43:59,781 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2158.67 MB 2025-02-14 08:43:59,781 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18524.14 MB 2025-02-14 08:43:59,781 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21988.64 MB 2025-02-14 08:43:59,781 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3464.50 MB 2025-02-14 08:43:59,781 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20929.31 MB 2025-02-14 08:43:59,910 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 08:43:59,910 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 08:43:59,910 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 08:43:59,910 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:43:59,910 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18833.73 MB 2025-02-14 08:43:59,910 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19234.49 MB 2025-02-14 08:43:59,910 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 400.76 MB 2025-02-14 08:43:59,910 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21988.64 MB 2025-02-14 08:43:59,910 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22202.55 MB 2025-02-14 08:43:59,910 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 213.91 MB 2025-02-14 08:43:59,910 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19606.67 MB 2025-02-14 08:43:59,926 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 08:43:59,927 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 08:43:59,927 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:43:59,927 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:43:59,927 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19450.23 MB 2025-02-14 08:43:59,927 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19679.63 MB 2025-02-14 08:43:59,927 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.40 MB 2025-02-14 08:43:59,927 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22202.55 MB 2025-02-14 08:43:59,927 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22202.55 MB 2025-02-14 08:43:59,927 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:43:59,927 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19736.13 MB 2025-02-14 08:43:59,929 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 08:43:59,929 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 08:43:59,929 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.86 seconds 2025-02-14 08:43:59,929 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:43:59,929 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13745.66 MB 2025-02-14 08:43:59,929 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19880.70 MB 2025-02-14 08:43:59,929 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6135.04 MB 2025-02-14 08:43:59,929 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54060.38 MB 2025-02-14 08:43:59,929 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22202.55 MB 2025-02-14 08:43:59,929 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31857.84 MB 2025-02-14 08:43:59,929 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19880.70 MB 2025-02-14 08:44:00,206 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 08:44:00,206 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 08:44:00,206 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 08:44:00,206 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:44:00,206 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14834.36 MB 2025-02-14 08:44:00,206 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17848.40 MB 2025-02-14 08:44:00,206 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-14 08:44:00,206 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22202.55 MB 2025-02-14 08:44:00,206 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22202.55 MB 2025-02-14 08:44:00,206 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:44:00,206 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18149.77 MB 2025-02-14 08:44:00,225 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 08:44:00,226 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 08:44:00,233 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 08:44:00,233 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 08:44:00,233 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:44:00,233 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:44:00,233 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17848.40 MB 2025-02-14 08:44:00,233 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26287.42 MB 2025-02-14 08:44:00,233 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 08:44:00,233 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22202.55 MB 2025-02-14 08:44:00,233 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32692.50 MB 2025-02-14 08:44:00,233 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 08:44:00,233 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26287.42 MB 2025-02-14 08:44:00,456 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 08:44:00,458 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:44:00,458 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 08:44:00,459 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:44:00,459 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 08:44:00,464 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 08:44:00,465 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:44:00,465 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 08:44:00,465 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 08:45:39,545 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:45:39,546 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 08:45:39,551 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 08:45:39,554 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:45:39,554 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2568, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 08:45:39,555 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:45:39,555 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2568, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 08:46:19,022 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 08:46:19,022 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 08:46:19,022 - resource_logging.py:150 - __exit__ - DEBUG - Time: 39.46 seconds 2025-02-14 08:46:19,023 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:46:19,023 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30865.80 MB 2025-02-14 08:46:19,023 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39954.86 MB 2025-02-14 08:46:19,023 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 9089.06 MB 2025-02-14 08:46:19,023 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63174.61 MB 2025-02-14 08:46:19,023 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43893.39 MB 2025-02-14 08:46:19,023 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19281.22 MB 2025-02-14 08:46:19,023 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49042.87 MB 2025-02-14 08:46:19,183 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 08:46:19,183 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 08:46:19,183 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 08:46:19,183 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:46:19,184 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39954.86 MB 2025-02-14 08:46:19,184 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29130.42 MB 2025-02-14 08:46:19,184 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10824.44 MB 2025-02-14 08:46:19,184 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43893.39 MB 2025-02-14 08:46:19,184 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 76118.23 MB 2025-02-14 08:46:19,184 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 32224.84 MB 2025-02-14 08:46:19,184 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 64612.99 MB 2025-02-14 08:46:21,158 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 08:46:21,158 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 08:46:21,158 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.97 seconds 2025-02-14 08:46:21,158 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:46:21,158 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29130.42 MB 2025-02-14 08:46:21,158 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29661.26 MB 2025-02-14 08:46:21,158 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 08:46:21,158 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 76118.23 MB 2025-02-14 08:46:21,158 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31675.38 MB 2025-02-14 08:46:21,158 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -44442.85 MB 2025-02-14 08:46:21,158 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33641.63 MB 2025-02-14 08:46:21,172 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 08:46:21,173 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 08:46:21,173 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:46:21,173 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:46:21,173 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29661.26 MB 2025-02-14 08:46:21,173 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31550.79 MB 2025-02-14 08:46:21,173 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 08:46:21,173 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31675.38 MB 2025-02-14 08:46:21,173 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34978.40 MB 2025-02-14 08:46:21,173 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 08:46:21,173 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32968.22 MB 2025-02-14 08:46:21,384 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 08:46:21,384 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 08:46:21,384 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 08:46:21,384 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:46:21,384 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31550.79 MB 2025-02-14 08:46:21,384 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33792.65 MB 2025-02-14 08:46:21,385 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 08:46:21,385 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34978.40 MB 2025-02-14 08:46:21,385 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41584.43 MB 2025-02-14 08:46:21,385 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 08:46:21,385 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39336.93 MB 2025-02-14 08:46:21,385 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 08:46:21,385 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 08:46:21,385 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 08:46:21,385 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:46:21,385 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29661.26 MB 2025-02-14 08:46:21,385 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33792.65 MB 2025-02-14 08:46:21,385 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 08:46:21,385 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31675.38 MB 2025-02-14 08:46:21,385 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41584.43 MB 2025-02-14 08:46:21,385 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9909.04 MB 2025-02-14 08:46:21,385 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39336.93 MB 2025-02-14 08:46:21,553 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 08:46:21,553 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 08:46:21,553 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 08:46:21,553 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:46:21,553 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35326.19 MB 2025-02-14 08:46:21,553 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36093.19 MB 2025-02-14 08:46:21,553 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 08:46:21,553 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41584.43 MB 2025-02-14 08:46:21,553 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42001.76 MB 2025-02-14 08:46:21,553 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 08:46:21,553 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36800.98 MB 2025-02-14 08:46:21,572 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 08:46:21,572 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 08:46:21,572 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:46:21,572 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:46:21,572 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36506.08 MB 2025-02-14 08:46:21,572 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36734.91 MB 2025-02-14 08:46:21,572 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.83 MB 2025-02-14 08:46:21,572 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42001.76 MB 2025-02-14 08:46:21,572 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42001.76 MB 2025-02-14 08:46:21,572 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:46:21,572 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36944.31 MB 2025-02-14 08:46:21,573 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 08:46:21,573 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 08:46:21,573 - resource_logging.py:150 - __exit__ - DEBUG - Time: 42.02 seconds 2025-02-14 08:46:21,573 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:46:21,573 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21917.25 MB 2025-02-14 08:46:21,573 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36935.29 MB 2025-02-14 08:46:21,573 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 15018.04 MB 2025-02-14 08:46:21,573 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54226.06 MB 2025-02-14 08:46:21,573 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42001.76 MB 2025-02-14 08:46:21,573 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12224.30 MB 2025-02-14 08:46:21,573 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36944.31 MB 2025-02-14 08:46:21,842 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 08:46:21,842 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 08:46:21,842 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 08:46:21,842 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:46:21,842 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36935.29 MB 2025-02-14 08:46:21,842 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26910.98 MB 2025-02-14 08:46:21,842 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10024.31 MB 2025-02-14 08:46:21,842 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42001.76 MB 2025-02-14 08:46:21,842 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42001.76 MB 2025-02-14 08:46:21,842 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:46:21,842 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39438.36 MB 2025-02-14 08:46:21,860 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8134, cut from 8136 2025-02-14 08:46:21,860 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 08:46:21,866 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 08:46:21,866 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 08:46:21,866 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:46:21,866 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:46:21,866 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26910.98 MB 2025-02-14 08:46:21,866 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35320.77 MB 2025-02-14 08:46:21,866 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8409.79 MB 2025-02-14 08:46:21,866 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42001.76 MB 2025-02-14 08:46:21,866 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46183.48 MB 2025-02-14 08:46:21,866 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4181.72 MB 2025-02-14 08:46:21,866 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35320.77 MB 2025-02-14 08:46:22,027 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7926] 2025-02-14 08:46:22,029 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:46:22,029 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 08:46:22,030 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:46:22,030 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 08:46:22,034 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 08:46:22,035 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:46:22,036 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 08:46:22,036 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 08:47:11,848 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:47:11,848 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 08:47:11,853 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 08:47:11,856 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:47:11,856 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2373, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 08:47:11,857 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:47:11,857 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2373, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 08:47:48,710 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 08:47:48,710 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 08:47:48,710 - resource_logging.py:150 - __exit__ - DEBUG - Time: 36.84 seconds 2025-02-14 08:47:48,710 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:47:48,710 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29504.15 MB 2025-02-14 08:47:48,710 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37902.06 MB 2025-02-14 08:47:48,710 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8397.91 MB 2025-02-14 08:47:48,710 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62925.05 MB 2025-02-14 08:47:48,710 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42498.79 MB 2025-02-14 08:47:48,710 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20426.26 MB 2025-02-14 08:47:48,710 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46903.56 MB 2025-02-14 08:47:48,865 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 08:47:48,865 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 08:47:48,865 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 08:47:48,865 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:47:48,865 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37902.06 MB 2025-02-14 08:47:48,865 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28114.34 MB 2025-02-14 08:47:48,865 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9787.73 MB 2025-02-14 08:47:48,865 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42498.79 MB 2025-02-14 08:47:48,865 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 74499.23 MB 2025-02-14 08:47:48,865 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 32000.44 MB 2025-02-14 08:47:48,865 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 62880.93 MB 2025-02-14 08:47:50,826 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 08:47:50,826 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 08:47:50,826 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.96 seconds 2025-02-14 08:47:50,826 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:47:50,826 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28114.34 MB 2025-02-14 08:47:50,826 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28645.18 MB 2025-02-14 08:47:50,826 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 08:47:50,826 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 74499.23 MB 2025-02-14 08:47:50,826 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31111.25 MB 2025-02-14 08:47:50,826 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -43387.98 MB 2025-02-14 08:47:50,826 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32625.55 MB 2025-02-14 08:47:50,840 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 08:47:50,840 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 08:47:50,840 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:47:50,840 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:47:50,840 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28645.18 MB 2025-02-14 08:47:50,840 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30534.25 MB 2025-02-14 08:47:50,840 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.08 MB 2025-02-14 08:47:50,840 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31111.25 MB 2025-02-14 08:47:50,840 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34414.26 MB 2025-02-14 08:47:50,840 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 08:47:50,840 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31951.68 MB 2025-02-14 08:47:51,049 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 08:47:51,049 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 08:47:51,049 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 08:47:51,049 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:47:51,049 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30534.25 MB 2025-02-14 08:47:51,049 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32776.11 MB 2025-02-14 08:47:51,049 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 08:47:51,049 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34414.26 MB 2025-02-14 08:47:51,049 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41020.29 MB 2025-02-14 08:47:51,049 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 08:47:51,049 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38320.39 MB 2025-02-14 08:47:51,050 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 08:47:51,050 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 08:47:51,050 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 08:47:51,050 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:47:51,050 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28645.18 MB 2025-02-14 08:47:51,050 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32776.11 MB 2025-02-14 08:47:51,050 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4130.93 MB 2025-02-14 08:47:51,050 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31111.25 MB 2025-02-14 08:47:51,050 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41020.29 MB 2025-02-14 08:47:51,050 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9909.04 MB 2025-02-14 08:47:51,050 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38320.39 MB 2025-02-14 08:47:51,218 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 08:47:51,218 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 08:47:51,218 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 08:47:51,218 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:47:51,218 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34309.65 MB 2025-02-14 08:47:51,218 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35076.65 MB 2025-02-14 08:47:51,218 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 08:47:51,218 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41020.29 MB 2025-02-14 08:47:51,218 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41433.43 MB 2025-02-14 08:47:51,218 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 08:47:51,218 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35784.44 MB 2025-02-14 08:47:51,237 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 08:47:51,237 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 08:47:51,237 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:47:51,237 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:47:51,237 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35489.54 MB 2025-02-14 08:47:51,237 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35718.17 MB 2025-02-14 08:47:51,237 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.63 MB 2025-02-14 08:47:51,237 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41433.43 MB 2025-02-14 08:47:51,237 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41433.43 MB 2025-02-14 08:47:51,237 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:47:51,237 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35965.40 MB 2025-02-14 08:47:51,238 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 08:47:51,238 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 08:47:51,238 - resource_logging.py:150 - __exit__ - DEBUG - Time: 39.38 seconds 2025-02-14 08:47:51,238 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:47:51,238 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21236.43 MB 2025-02-14 08:47:51,238 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35918.61 MB 2025-02-14 08:47:51,238 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14682.18 MB 2025-02-14 08:47:51,238 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62925.05 MB 2025-02-14 08:47:51,238 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41433.43 MB 2025-02-14 08:47:51,238 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21491.61 MB 2025-02-14 08:47:51,238 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35965.40 MB 2025-02-14 08:47:51,509 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 08:47:51,509 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 08:47:51,509 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 08:47:51,509 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:47:51,509 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35918.61 MB 2025-02-14 08:47:51,509 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26230.91 MB 2025-02-14 08:47:51,509 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9687.69 MB 2025-02-14 08:47:51,509 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41433.43 MB 2025-02-14 08:47:51,509 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41433.43 MB 2025-02-14 08:47:51,509 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:47:51,509 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38422.29 MB 2025-02-14 08:47:51,527 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8136, cut from 8138 2025-02-14 08:47:51,527 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 08:47:51,534 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 08:47:51,534 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 08:47:51,534 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:47:51,534 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:47:51,534 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26230.91 MB 2025-02-14 08:47:51,534 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34643.34 MB 2025-02-14 08:47:51,534 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8412.43 MB 2025-02-14 08:47:51,534 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41433.43 MB 2025-02-14 08:47:51,534 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49796.87 MB 2025-02-14 08:47:51,534 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8363.44 MB 2025-02-14 08:47:51,534 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34643.34 MB 2025-02-14 08:47:51,700 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7928] 2025-02-14 08:47:51,701 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:47:51,701 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 08:47:51,702 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:47:51,702 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 08:47:51,707 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 08:47:51,708 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:47:51,708 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 08:47:51,708 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 08:48:40,566 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:48:40,566 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 08:48:40,571 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 08:48:40,574 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:48:40,574 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1053, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 08:48:40,575 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:48:40,575 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1053, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 08:48:56,882 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 08:48:56,882 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 08:48:56,882 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.30 seconds 2025-02-14 08:48:56,882 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:48:56,882 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20306.18 MB 2025-02-14 08:48:56,882 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24032.82 MB 2025-02-14 08:48:56,882 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3726.64 MB 2025-02-14 08:48:56,882 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58160.32 MB 2025-02-14 08:48:56,882 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28995.22 MB 2025-02-14 08:48:56,882 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29165.09 MB 2025-02-14 08:48:56,882 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32949.25 MB 2025-02-14 08:48:56,952 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 08:48:56,952 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 08:48:56,952 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 08:48:56,952 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:48:56,952 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24032.82 MB 2025-02-14 08:48:56,952 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21252.07 MB 2025-02-14 08:48:56,952 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2780.75 MB 2025-02-14 08:48:56,952 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28995.22 MB 2025-02-14 08:48:56,952 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41846.57 MB 2025-02-14 08:48:56,952 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 12851.35 MB 2025-02-14 08:48:56,952 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35621.75 MB 2025-02-14 08:48:58,884 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 08:48:58,884 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 08:48:58,884 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 08:48:58,884 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:48:58,884 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21252.07 MB 2025-02-14 08:48:58,884 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21782.91 MB 2025-02-14 08:48:58,884 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 08:48:58,884 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41846.57 MB 2025-02-14 08:48:58,884 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26684.16 MB 2025-02-14 08:48:58,884 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15162.41 MB 2025-02-14 08:48:58,884 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25763.13 MB 2025-02-14 08:48:58,898 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 08:48:58,898 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 08:48:58,898 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:48:58,898 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:48:58,898 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21782.91 MB 2025-02-14 08:48:58,898 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23672.44 MB 2025-02-14 08:48:58,898 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 08:48:58,898 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26684.16 MB 2025-02-14 08:48:58,898 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28571.60 MB 2025-02-14 08:48:58,898 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 08:48:58,898 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25089.87 MB 2025-02-14 08:48:59,109 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 08:48:59,109 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 08:48:59,109 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 08:48:59,109 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:48:59,109 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23672.44 MB 2025-02-14 08:48:59,109 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25914.30 MB 2025-02-14 08:48:59,109 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 08:48:59,109 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28571.60 MB 2025-02-14 08:48:59,109 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34233.91 MB 2025-02-14 08:48:59,109 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 08:48:59,109 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31458.58 MB 2025-02-14 08:48:59,109 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 08:48:59,109 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 08:48:59,109 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 08:48:59,109 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:48:59,109 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21782.91 MB 2025-02-14 08:48:59,109 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25914.30 MB 2025-02-14 08:48:59,110 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 08:48:59,110 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26684.16 MB 2025-02-14 08:48:59,110 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34233.91 MB 2025-02-14 08:48:59,110 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-14 08:48:59,110 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31458.58 MB 2025-02-14 08:48:59,316 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 08:48:59,316 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 08:48:59,316 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 08:48:59,316 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:48:59,316 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27447.84 MB 2025-02-14 08:48:59,316 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28214.84 MB 2025-02-14 08:48:59,316 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 08:48:59,316 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34233.91 MB 2025-02-14 08:48:59,316 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34651.24 MB 2025-02-14 08:48:59,316 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 08:48:59,316 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28922.63 MB 2025-02-14 08:48:59,335 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 08:48:59,335 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 08:48:59,335 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:48:59,335 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:48:59,335 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28627.73 MB 2025-02-14 08:48:59,335 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28855.59 MB 2025-02-14 08:48:59,335 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.85 MB 2025-02-14 08:48:59,335 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34651.24 MB 2025-02-14 08:48:59,335 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34651.24 MB 2025-02-14 08:48:59,335 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:48:59,335 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29096.23 MB 2025-02-14 08:48:59,336 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 08:48:59,336 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 08:48:59,336 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.76 seconds 2025-02-14 08:48:59,336 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:48:59,336 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16637.44 MB 2025-02-14 08:48:59,336 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29056.56 MB 2025-02-14 08:48:59,336 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12419.12 MB 2025-02-14 08:48:59,336 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58160.32 MB 2025-02-14 08:48:59,336 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34651.24 MB 2025-02-14 08:48:59,336 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23509.07 MB 2025-02-14 08:48:59,336 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29096.23 MB 2025-02-14 08:48:59,604 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 08:48:59,604 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 08:48:59,604 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 08:48:59,604 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:48:59,605 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29056.56 MB 2025-02-14 08:48:59,605 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21640.31 MB 2025-02-14 08:48:59,605 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7416.25 MB 2025-02-14 08:48:59,605 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34651.24 MB 2025-02-14 08:48:59,605 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34651.24 MB 2025-02-14 08:48:59,605 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:48:59,605 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31567.00 MB 2025-02-14 08:48:59,623 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8158, cut from 8160 2025-02-14 08:48:59,623 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 08:48:59,629 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 08:48:59,629 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 08:48:59,629 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:48:59,629 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:48:59,629 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21640.31 MB 2025-02-14 08:48:59,629 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30075.16 MB 2025-02-14 08:48:59,629 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8434.85 MB 2025-02-14 08:48:59,629 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34651.24 MB 2025-02-14 08:48:59,629 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45134.91 MB 2025-02-14 08:48:59,629 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10483.66 MB 2025-02-14 08:48:59,629 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30075.16 MB 2025-02-14 08:48:59,791 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7950] 2025-02-14 08:48:59,793 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:48:59,793 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 08:48:59,794 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:48:59,794 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 08:48:59,799 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 08:48:59,800 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:48:59,800 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 08:48:59,800 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 08:49:49,161 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:49:49,161 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 08:49:49,166 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 08:49:49,169 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:49:49,169 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1003, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 08:49:49,170 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:49:49,170 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1003, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 08:50:04,601 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 08:50:04,601 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 08:50:04,601 - resource_logging.py:150 - __exit__ - DEBUG - Time: 15.42 seconds 2025-02-14 08:50:04,601 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:50:04,601 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19957.77 MB 2025-02-14 08:50:04,601 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23508.25 MB 2025-02-14 08:50:04,601 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3550.48 MB 2025-02-14 08:50:04,601 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57713.62 MB 2025-02-14 08:50:04,601 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30918.31 MB 2025-02-14 08:50:04,601 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26795.31 MB 2025-02-14 08:50:04,601 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32374.35 MB 2025-02-14 08:50:04,661 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 08:50:04,661 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 08:50:04,661 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 08:50:04,661 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:50:04,661 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23508.25 MB 2025-02-14 08:50:04,661 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20992.14 MB 2025-02-14 08:50:04,661 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2516.11 MB 2025-02-14 08:50:04,661 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30918.31 MB 2025-02-14 08:50:04,661 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39093.01 MB 2025-02-14 08:50:04,661 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8174.70 MB 2025-02-14 08:50:04,661 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34529.77 MB 2025-02-14 08:50:06,568 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 08:50:06,568 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 08:50:06,568 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 08:50:06,568 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:50:06,568 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20992.14 MB 2025-02-14 08:50:06,568 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21522.98 MB 2025-02-14 08:50:06,569 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 08:50:06,569 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39093.01 MB 2025-02-14 08:50:06,569 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28783.41 MB 2025-02-14 08:50:06,569 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10309.60 MB 2025-02-14 08:50:06,569 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25502.31 MB 2025-02-14 08:50:06,582 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 08:50:06,582 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 08:50:06,582 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:50:06,582 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:50:06,582 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21522.98 MB 2025-02-14 08:50:06,582 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23412.51 MB 2025-02-14 08:50:06,582 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 08:50:06,582 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28783.41 MB 2025-02-14 08:50:06,582 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28783.41 MB 2025-02-14 08:50:06,582 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:50:06,582 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24829.94 MB 2025-02-14 08:50:06,798 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 08:50:06,798 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 08:50:06,798 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 08:50:06,798 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:50:06,798 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23412.51 MB 2025-02-14 08:50:06,798 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25654.37 MB 2025-02-14 08:50:06,798 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 08:50:06,798 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28783.41 MB 2025-02-14 08:50:06,798 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34445.72 MB 2025-02-14 08:50:06,798 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 08:50:06,798 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31198.65 MB 2025-02-14 08:50:06,799 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 08:50:06,799 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 08:50:06,799 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 08:50:06,799 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:50:06,799 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21522.98 MB 2025-02-14 08:50:06,799 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25654.37 MB 2025-02-14 08:50:06,799 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 08:50:06,799 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28783.41 MB 2025-02-14 08:50:06,799 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34445.72 MB 2025-02-14 08:50:06,799 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 08:50:06,799 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31198.65 MB 2025-02-14 08:50:06,973 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 08:50:06,973 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 08:50:06,973 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 08:50:06,973 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:50:06,973 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27187.91 MB 2025-02-14 08:50:06,973 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27954.91 MB 2025-02-14 08:50:06,973 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 08:50:06,973 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34445.72 MB 2025-02-14 08:50:06,973 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34860.96 MB 2025-02-14 08:50:06,973 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 08:50:06,973 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28662.70 MB 2025-02-14 08:50:06,992 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 08:50:06,992 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 08:50:06,992 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:50:06,992 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:50:06,992 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28367.80 MB 2025-02-14 08:50:06,992 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28596.27 MB 2025-02-14 08:50:06,992 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.47 MB 2025-02-14 08:50:06,992 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34860.96 MB 2025-02-14 08:50:06,992 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34860.96 MB 2025-02-14 08:50:06,992 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:50:06,992 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28783.18 MB 2025-02-14 08:50:06,993 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 08:50:06,993 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 08:50:06,993 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.82 seconds 2025-02-14 08:50:06,993 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:50:06,993 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16463.24 MB 2025-02-14 08:50:06,993 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28796.14 MB 2025-02-14 08:50:06,993 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12332.90 MB 2025-02-14 08:50:06,993 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57713.62 MB 2025-02-14 08:50:06,993 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34860.96 MB 2025-02-14 08:50:06,993 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22852.67 MB 2025-02-14 08:50:06,993 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28796.14 MB 2025-02-14 08:50:07,259 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 08:50:07,259 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 08:50:07,259 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 08:50:07,259 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:50:07,259 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28796.14 MB 2025-02-14 08:50:07,259 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21448.96 MB 2025-02-14 08:50:07,259 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7347.17 MB 2025-02-14 08:50:07,259 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34860.96 MB 2025-02-14 08:50:07,259 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34860.96 MB 2025-02-14 08:50:07,259 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:50:07,259 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31292.75 MB 2025-02-14 08:50:07,277 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8113, cut from 8115 2025-02-14 08:50:07,277 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 08:50:07,283 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 08:50:07,283 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 08:50:07,283 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:50:07,283 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:50:07,283 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21448.96 MB 2025-02-14 08:50:07,283 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29837.38 MB 2025-02-14 08:50:07,283 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8388.42 MB 2025-02-14 08:50:07,283 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34860.96 MB 2025-02-14 08:50:07,283 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43201.33 MB 2025-02-14 08:50:07,283 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8340.37 MB 2025-02-14 08:50:07,283 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29837.38 MB 2025-02-14 08:50:07,444 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7905] 2025-02-14 08:50:07,445 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:50:07,445 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 08:50:07,446 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:50:07,446 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 08:50:07,451 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 08:50:07,452 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:50:07,452 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 08:50:07,452 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 08:50:15,908 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:50:15,908 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 08:50:15,913 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 08:50:15,916 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:50:15,916 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1129, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 08:50:15,917 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:50:15,917 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1129, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 08:50:33,495 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 08:50:33,495 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 08:50:33,495 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.57 seconds 2025-02-14 08:50:33,495 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:50:33,495 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20835.76 MB 2025-02-14 08:50:33,496 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24831.23 MB 2025-02-14 08:50:33,496 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3995.47 MB 2025-02-14 08:50:33,496 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55710.84 MB 2025-02-14 08:50:33,496 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31352.42 MB 2025-02-14 08:50:33,496 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24358.42 MB 2025-02-14 08:50:33,496 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33705.32 MB 2025-02-14 08:50:33,603 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 08:50:33,603 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 08:50:33,603 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 08:50:33,603 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:50:33,603 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24831.23 MB 2025-02-14 08:50:33,603 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21647.17 MB 2025-02-14 08:50:33,603 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3184.06 MB 2025-02-14 08:50:33,603 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31352.42 MB 2025-02-14 08:50:33,603 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41915.78 MB 2025-02-14 08:50:33,603 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10563.35 MB 2025-02-14 08:50:33,603 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36793.19 MB 2025-02-14 08:50:35,538 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 08:50:35,538 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 08:50:35,538 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 08:50:35,538 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:50:35,538 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21647.17 MB 2025-02-14 08:50:35,538 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22178.01 MB 2025-02-14 08:50:35,538 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 08:50:35,538 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41915.78 MB 2025-02-14 08:50:35,538 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26673.68 MB 2025-02-14 08:50:35,538 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15242.10 MB 2025-02-14 08:50:35,538 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26157.34 MB 2025-02-14 08:50:35,552 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 08:50:35,552 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 08:50:35,552 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:50:35,552 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:50:35,552 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22178.01 MB 2025-02-14 08:50:35,552 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24067.55 MB 2025-02-14 08:50:35,552 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 08:50:35,552 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26673.68 MB 2025-02-14 08:50:35,552 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28561.11 MB 2025-02-14 08:50:35,552 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 08:50:35,552 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25484.97 MB 2025-02-14 08:50:35,763 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 08:50:35,763 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 08:50:35,763 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 08:50:35,763 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:50:35,763 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24067.55 MB 2025-02-14 08:50:35,763 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26309.40 MB 2025-02-14 08:50:35,763 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 08:50:35,763 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28561.11 MB 2025-02-14 08:50:35,763 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34223.42 MB 2025-02-14 08:50:35,763 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 08:50:35,763 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31853.68 MB 2025-02-14 08:50:35,764 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 08:50:35,764 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 08:50:35,764 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 08:50:35,764 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:50:35,764 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22178.01 MB 2025-02-14 08:50:35,764 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26309.40 MB 2025-02-14 08:50:35,764 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 08:50:35,764 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26673.68 MB 2025-02-14 08:50:35,764 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34223.42 MB 2025-02-14 08:50:35,764 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-14 08:50:35,764 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31853.68 MB 2025-02-14 08:50:35,930 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 08:50:35,930 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 08:50:35,930 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 08:50:35,930 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:50:35,930 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27842.94 MB 2025-02-14 08:50:35,930 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28609.95 MB 2025-02-14 08:50:35,930 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 08:50:35,930 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34223.42 MB 2025-02-14 08:50:35,930 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34638.66 MB 2025-02-14 08:50:35,930 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 08:50:35,930 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29317.73 MB 2025-02-14 08:50:35,949 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 08:50:35,949 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 08:50:35,949 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:50:35,949 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:50:35,949 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29022.83 MB 2025-02-14 08:50:35,949 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29250.34 MB 2025-02-14 08:50:35,949 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.51 MB 2025-02-14 08:50:35,949 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34638.66 MB 2025-02-14 08:50:35,949 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34638.66 MB 2025-02-14 08:50:35,949 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:50:35,949 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29468.98 MB 2025-02-14 08:50:35,950 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 08:50:35,950 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 08:50:35,950 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.03 seconds 2025-02-14 08:50:35,950 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:50:35,950 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16902.23 MB 2025-02-14 08:50:35,950 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29450.46 MB 2025-02-14 08:50:35,950 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12548.23 MB 2025-02-14 08:50:35,950 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55710.84 MB 2025-02-14 08:50:35,950 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34638.66 MB 2025-02-14 08:50:35,950 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21072.18 MB 2025-02-14 08:50:35,950 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29468.98 MB 2025-02-14 08:50:36,221 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 08:50:36,221 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 08:50:36,221 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 08:50:36,221 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:50:36,221 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29450.46 MB 2025-02-14 08:50:36,221 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21891.77 MB 2025-02-14 08:50:36,221 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7558.69 MB 2025-02-14 08:50:36,221 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34638.66 MB 2025-02-14 08:50:36,221 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34638.66 MB 2025-02-14 08:50:36,221 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:50:36,221 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31950.15 MB 2025-02-14 08:50:36,239 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8123, cut from 8125 2025-02-14 08:50:36,239 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 08:50:36,245 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 08:50:36,245 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 08:50:36,245 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:50:36,245 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:50:36,245 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21891.77 MB 2025-02-14 08:50:36,245 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30291.15 MB 2025-02-14 08:50:36,245 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8399.39 MB 2025-02-14 08:50:36,245 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34638.66 MB 2025-02-14 08:50:36,245 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42989.52 MB 2025-02-14 08:50:36,245 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8350.86 MB 2025-02-14 08:50:36,245 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30291.15 MB 2025-02-14 08:50:36,407 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7915] 2025-02-14 08:50:36,408 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:50:36,408 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 08:50:36,409 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:50:36,409 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 08:50:36,414 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 08:50:36,415 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:50:36,415 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 08:50:36,415 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 08:51:47,950 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:51:47,950 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 08:51:47,959 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 08:51:47,965 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:51:47,965 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 140, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 08:51:47,967 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:51:47,967 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 140, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 08:51:50,230 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 08:51:50,230 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 08:51:50,230 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.26 seconds 2025-02-14 08:51:50,230 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:51:50,230 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13944.25 MB 2025-02-14 08:51:50,230 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14439.70 MB 2025-02-14 08:51:50,230 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 495.45 MB 2025-02-14 08:51:50,230 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51340.38 MB 2025-02-14 08:51:50,230 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17853.05 MB 2025-02-14 08:51:50,230 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33487.32 MB 2025-02-14 08:51:50,230 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23416.43 MB 2025-02-14 08:51:50,245 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 08:51:50,245 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 08:51:50,245 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:51:50,245 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:51:50,245 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14439.70 MB 2025-02-14 08:51:50,245 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14679.75 MB 2025-02-14 08:51:50,245 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 240.05 MB 2025-02-14 08:51:50,245 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17853.05 MB 2025-02-14 08:51:50,245 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17853.05 MB 2025-02-14 08:51:50,245 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:51:50,245 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16448.68 MB 2025-02-14 08:51:50,958 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 08:51:50,959 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 08:51:50,959 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.71 seconds 2025-02-14 08:51:50,959 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:51:50,959 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14679.75 MB 2025-02-14 08:51:50,959 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14865.54 MB 2025-02-14 08:51:50,959 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 185.79 MB 2025-02-14 08:51:50,959 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17853.05 MB 2025-02-14 08:51:50,959 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17853.05 MB 2025-02-14 08:51:50,959 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:51:50,959 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18851.22 MB 2025-02-14 08:51:50,969 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 08:51:50,969 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 08:51:50,969 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 08:51:50,969 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:51:50,969 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14865.48 MB 2025-02-14 08:51:50,969 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15526.65 MB 2025-02-14 08:51:50,969 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 661.18 MB 2025-02-14 08:51:50,969 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17853.05 MB 2025-02-14 08:51:50,969 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17853.05 MB 2025-02-14 08:51:50,969 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:51:50,969 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16022.76 MB 2025-02-14 08:51:51,077 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 08:51:51,077 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 08:51:51,077 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 08:51:51,077 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:51:51,077 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15526.65 MB 2025-02-14 08:51:51,077 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16311.34 MB 2025-02-14 08:51:51,077 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 784.69 MB 2025-02-14 08:51:51,077 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17853.05 MB 2025-02-14 08:51:51,077 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19344.13 MB 2025-02-14 08:51:51,077 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1491.08 MB 2025-02-14 08:51:51,077 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18256.52 MB 2025-02-14 08:51:51,078 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 08:51:51,078 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 08:51:51,078 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 08:51:51,078 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:51:51,078 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14865.48 MB 2025-02-14 08:51:51,078 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16311.34 MB 2025-02-14 08:51:51,078 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1445.87 MB 2025-02-14 08:51:51,078 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17853.05 MB 2025-02-14 08:51:51,078 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19344.13 MB 2025-02-14 08:51:51,078 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1491.08 MB 2025-02-14 08:51:51,078 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18256.52 MB 2025-02-14 08:51:51,182 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 08:51:51,182 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 08:51:51,182 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 08:51:51,183 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:51:51,183 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16848.08 MB 2025-02-14 08:51:51,183 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17117.06 MB 2025-02-14 08:51:51,183 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 268.98 MB 2025-02-14 08:51:51,183 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19344.13 MB 2025-02-14 08:51:51,183 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19486.74 MB 2025-02-14 08:51:51,183 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 142.61 MB 2025-02-14 08:51:51,183 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17375.17 MB 2025-02-14 08:51:51,198 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 08:51:51,198 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 08:51:51,198 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:51:51,198 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:51:51,198 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17261.58 MB 2025-02-14 08:51:51,198 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17471.98 MB 2025-02-14 08:51:51,198 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 210.41 MB 2025-02-14 08:51:51,198 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19486.74 MB 2025-02-14 08:51:51,198 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19486.74 MB 2025-02-14 08:51:51,198 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:51:51,198 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17480.08 MB 2025-02-14 08:51:51,200 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 08:51:51,200 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 08:51:51,201 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.23 seconds 2025-02-14 08:51:51,201 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:51:51,201 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13456.48 MB 2025-02-14 08:51:51,201 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17672.81 MB 2025-02-14 08:51:51,201 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4216.33 MB 2025-02-14 08:51:51,201 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51340.38 MB 2025-02-14 08:51:51,201 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19486.74 MB 2025-02-14 08:51:51,201 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31853.64 MB 2025-02-14 08:51:51,201 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17672.81 MB 2025-02-14 08:51:51,496 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 08:51:51,496 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 08:51:51,496 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-14 08:51:51,496 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:51:51,496 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17672.81 MB 2025-02-14 08:51:51,496 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20683.16 MB 2025-02-14 08:51:51,497 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3010.35 MB 2025-02-14 08:51:51,497 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19486.74 MB 2025-02-14 08:51:51,497 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22573.74 MB 2025-02-14 08:51:51,497 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3087.01 MB 2025-02-14 08:51:51,497 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20984.58 MB 2025-02-14 08:51:51,516 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8152, cut from 8154 2025-02-14 08:51:51,517 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2,'] 2025-02-14 08:51:51,524 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 08:51:51,524 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 08:51:51,524 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:51:51,524 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:51:51,524 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20683.16 MB 2025-02-14 08:51:51,524 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29112.28 MB 2025-02-14 08:51:51,524 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8429.12 MB 2025-02-14 08:51:51,524 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22573.74 MB 2025-02-14 08:51:51,524 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33049.02 MB 2025-02-14 08:51:51,524 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10475.27 MB 2025-02-14 08:51:51,524 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29112.28 MB 2025-02-14 08:51:51,791 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7944] 2025-02-14 08:51:51,793 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:51:51,794 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 08:51:51,796 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:51:51,796 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 08:51:51,804 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 08:51:51,807 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:51:51,807 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 08:51:51,807 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2,'] 2025-02-14 08:52:38,236 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:52:38,236 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 08:52:38,241 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 08:52:38,245 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:52:38,245 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1694, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 08:52:38,246 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:52:38,246 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1694, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 08:53:04,402 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 08:53:04,402 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 08:53:04,402 - resource_logging.py:150 - __exit__ - DEBUG - Time: 26.15 seconds 2025-02-14 08:53:04,402 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:53:04,402 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33602.28 MB 2025-02-14 08:53:04,402 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39598.04 MB 2025-02-14 08:53:04,402 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5995.76 MB 2025-02-14 08:53:04,402 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44335.89 MB 2025-02-14 08:53:04,402 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44484.79 MB 2025-02-14 08:53:04,402 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 148.90 MB 2025-02-14 08:53:04,402 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48510.28 MB 2025-02-14 08:53:04,498 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 08:53:04,498 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 08:53:04,498 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 08:53:04,498 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:53:04,498 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39598.04 MB 2025-02-14 08:53:04,498 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33413.94 MB 2025-02-14 08:53:04,498 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6184.10 MB 2025-02-14 08:53:04,498 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44484.79 MB 2025-02-14 08:53:04,498 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 60978.89 MB 2025-02-14 08:53:04,498 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 16494.10 MB 2025-02-14 08:53:04,498 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 53334.17 MB 2025-02-14 08:53:06,450 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 08:53:06,450 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 08:53:06,450 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.95 seconds 2025-02-14 08:53:06,450 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:53:06,450 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33413.94 MB 2025-02-14 08:53:06,450 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33944.78 MB 2025-02-14 08:53:06,450 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 08:53:06,450 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60978.89 MB 2025-02-14 08:53:06,450 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35714.50 MB 2025-02-14 08:53:06,450 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25264.39 MB 2025-02-14 08:53:06,450 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37926.19 MB 2025-02-14 08:53:06,464 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 08:53:06,464 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 08:53:06,464 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:53:06,464 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:53:06,464 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33944.78 MB 2025-02-14 08:53:06,464 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35834.32 MB 2025-02-14 08:53:06,464 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 08:53:06,464 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35714.50 MB 2025-02-14 08:53:06,464 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39019.61 MB 2025-02-14 08:53:06,464 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3305.11 MB 2025-02-14 08:53:06,464 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37251.75 MB 2025-02-14 08:53:06,675 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 08:53:06,675 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 08:53:06,675 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 08:53:06,675 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:53:06,675 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35834.32 MB 2025-02-14 08:53:06,676 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29246.67 MB 2025-02-14 08:53:06,676 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6587.65 MB 2025-02-14 08:53:06,676 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39019.61 MB 2025-02-14 08:53:06,676 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39963.33 MB 2025-02-14 08:53:06,676 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 08:53:06,676 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36166.88 MB 2025-02-14 08:53:06,676 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 08:53:06,676 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 08:53:06,676 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 08:53:06,676 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:53:06,676 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33944.78 MB 2025-02-14 08:53:06,676 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29246.67 MB 2025-02-14 08:53:06,676 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4698.12 MB 2025-02-14 08:53:06,676 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35714.50 MB 2025-02-14 08:53:06,676 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39963.33 MB 2025-02-14 08:53:06,676 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4248.83 MB 2025-02-14 08:53:06,676 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36166.88 MB 2025-02-14 08:53:06,844 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 08:53:06,845 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 08:53:06,845 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 08:53:06,845 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:53:06,845 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30780.21 MB 2025-02-14 08:53:06,845 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31547.21 MB 2025-02-14 08:53:06,845 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 08:53:06,845 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39963.33 MB 2025-02-14 08:53:06,845 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40233.86 MB 2025-02-14 08:53:06,845 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 270.53 MB 2025-02-14 08:53:06,845 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32255.00 MB 2025-02-14 08:53:06,864 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 08:53:06,864 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 08:53:06,864 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:53:06,864 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:53:06,864 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31960.10 MB 2025-02-14 08:53:06,864 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32188.77 MB 2025-02-14 08:53:06,864 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.67 MB 2025-02-14 08:53:06,864 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40233.86 MB 2025-02-14 08:53:06,864 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40233.86 MB 2025-02-14 08:53:06,864 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:53:06,864 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32425.16 MB 2025-02-14 08:53:06,865 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 08:53:06,865 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 08:53:06,865 - resource_logging.py:150 - __exit__ - DEBUG - Time: 28.62 seconds 2025-02-14 08:53:06,865 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:53:06,865 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27700.24 MB 2025-02-14 08:53:06,865 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32389.35 MB 2025-02-14 08:53:06,865 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4689.11 MB 2025-02-14 08:53:06,865 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41431.33 MB 2025-02-14 08:53:06,865 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40233.86 MB 2025-02-14 08:53:06,865 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1197.47 MB 2025-02-14 08:53:06,865 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32425.16 MB 2025-02-14 08:53:07,134 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 08:53:07,134 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 08:53:07,134 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 08:53:07,134 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:53:07,134 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32389.35 MB 2025-02-14 08:53:07,134 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23867.51 MB 2025-02-14 08:53:07,134 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8521.84 MB 2025-02-14 08:53:07,134 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40233.86 MB 2025-02-14 08:53:07,134 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40233.86 MB 2025-02-14 08:53:07,134 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:53:07,134 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34894.87 MB 2025-02-14 08:53:07,151 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8142, cut from 8144 2025-02-14 08:53:07,152 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 08:53:07,158 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 08:53:07,158 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 08:53:07,158 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:53:07,158 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:53:07,158 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23867.51 MB 2025-02-14 08:53:07,158 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32285.66 MB 2025-02-14 08:53:07,158 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8418.15 MB 2025-02-14 08:53:07,158 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40233.86 MB 2025-02-14 08:53:07,158 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44417.68 MB 2025-02-14 08:53:07,158 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4183.82 MB 2025-02-14 08:53:07,158 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32285.66 MB 2025-02-14 08:53:07,330 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7934] 2025-02-14 08:53:07,331 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:53:07,331 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 08:53:07,332 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:53:07,332 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 08:53:07,337 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 08:53:07,338 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:53:07,338 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 08:53:07,339 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 08:53:16,574 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:53:16,574 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 08:53:16,579 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 08:53:16,582 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:53:16,582 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1230, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 08:53:16,583 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:53:16,583 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1230, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 08:53:35,721 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 08:53:35,721 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 08:53:35,721 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.13 seconds 2025-02-14 08:53:35,721 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:53:35,721 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21539.54 MB 2025-02-14 08:53:35,721 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25893.23 MB 2025-02-14 08:53:35,721 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4353.69 MB 2025-02-14 08:53:35,721 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56971.23 MB 2025-02-14 08:53:35,721 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36721.13 MB 2025-02-14 08:53:35,721 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20250.10 MB 2025-02-14 08:53:35,721 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34861.28 MB 2025-02-14 08:53:35,793 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 08:53:35,793 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 08:53:35,793 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 08:53:35,793 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:53:35,793 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25893.23 MB 2025-02-14 08:53:35,793 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22172.24 MB 2025-02-14 08:53:35,793 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3720.99 MB 2025-02-14 08:53:35,793 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36721.13 MB 2025-02-14 08:53:35,793 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45361.40 MB 2025-02-14 08:53:35,793 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8640.27 MB 2025-02-14 08:53:35,793 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38935.98 MB 2025-02-14 08:53:37,729 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 08:53:37,729 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 08:53:37,729 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 08:53:37,729 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:53:37,729 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22172.24 MB 2025-02-14 08:53:37,729 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22703.08 MB 2025-02-14 08:53:37,729 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 08:53:37,729 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45361.40 MB 2025-02-14 08:53:37,729 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28183.63 MB 2025-02-14 08:53:37,729 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17177.77 MB 2025-02-14 08:53:37,729 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26682.41 MB 2025-02-14 08:53:37,745 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 08:53:37,745 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 08:53:37,745 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:53:37,745 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:53:37,745 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22703.08 MB 2025-02-14 08:53:37,745 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24592.61 MB 2025-02-14 08:53:37,745 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 08:53:37,745 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28183.63 MB 2025-02-14 08:53:37,745 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29127.34 MB 2025-02-14 08:53:37,745 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 08:53:37,745 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26010.04 MB 2025-02-14 08:53:37,958 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 08:53:37,958 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 08:53:37,958 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 08:53:37,958 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:53:37,958 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24592.61 MB 2025-02-14 08:53:37,958 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26834.47 MB 2025-02-14 08:53:37,958 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 08:53:37,958 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29127.34 MB 2025-02-14 08:53:37,958 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34789.65 MB 2025-02-14 08:53:37,958 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 08:53:37,958 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32378.75 MB 2025-02-14 08:53:37,959 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 08:53:37,959 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 08:53:37,959 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 08:53:37,959 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:53:37,959 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22703.08 MB 2025-02-14 08:53:37,959 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26834.47 MB 2025-02-14 08:53:37,959 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 08:53:37,959 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28183.63 MB 2025-02-14 08:53:37,959 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34789.65 MB 2025-02-14 08:53:37,959 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 08:53:37,959 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32378.75 MB 2025-02-14 08:53:38,125 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 08:53:38,125 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 08:53:38,125 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 08:53:38,125 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:53:38,125 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28368.01 MB 2025-02-14 08:53:38,125 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29135.01 MB 2025-02-14 08:53:38,125 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 08:53:38,125 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34789.65 MB 2025-02-14 08:53:38,125 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35202.79 MB 2025-02-14 08:53:38,125 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 08:53:38,125 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29842.80 MB 2025-02-14 08:53:38,144 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 08:53:38,144 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 08:53:38,144 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:53:38,144 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:53:38,144 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29547.90 MB 2025-02-14 08:53:38,144 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29776.80 MB 2025-02-14 08:53:38,144 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.90 MB 2025-02-14 08:53:38,144 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35202.79 MB 2025-02-14 08:53:38,144 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35202.79 MB 2025-02-14 08:53:38,144 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:53:38,144 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30012.02 MB 2025-02-14 08:53:38,145 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 08:53:38,145 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 08:53:38,145 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.56 seconds 2025-02-14 08:53:38,145 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:53:38,145 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17254.12 MB 2025-02-14 08:53:38,145 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29977.38 MB 2025-02-14 08:53:38,145 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12723.25 MB 2025-02-14 08:53:38,145 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56971.23 MB 2025-02-14 08:53:38,145 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35202.79 MB 2025-02-14 08:53:38,145 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21768.44 MB 2025-02-14 08:53:38,145 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30012.02 MB 2025-02-14 08:53:38,415 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 08:53:38,415 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 08:53:38,415 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 08:53:38,415 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:53:38,415 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29977.38 MB 2025-02-14 08:53:38,415 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22250.90 MB 2025-02-14 08:53:38,415 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7726.48 MB 2025-02-14 08:53:38,415 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35202.79 MB 2025-02-14 08:53:38,415 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35202.79 MB 2025-02-14 08:53:38,415 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:53:38,415 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32482.90 MB 2025-02-14 08:53:38,433 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8142, cut from 8144 2025-02-14 08:53:38,433 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 08:53:38,439 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 08:53:38,440 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 08:53:38,440 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:53:38,440 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:53:38,440 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22250.90 MB 2025-02-14 08:53:38,440 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30669.05 MB 2025-02-14 08:53:38,440 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8418.15 MB 2025-02-14 08:53:38,440 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35202.79 MB 2025-02-14 08:53:38,440 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43572.53 MB 2025-02-14 08:53:38,440 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8369.73 MB 2025-02-14 08:53:38,440 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30669.05 MB 2025-02-14 08:53:38,602 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7934] 2025-02-14 08:53:38,604 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:53:38,604 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 08:53:38,605 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:53:38,605 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 08:53:38,609 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 08:53:38,610 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:53:38,610 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 08:53:38,611 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 08:55:18,195 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:55:18,195 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 08:55:18,200 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 08:55:18,204 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:55:18,204 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 161, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 08:55:18,206 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:55:18,206 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 161, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 08:55:20,694 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 08:55:20,694 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 08:55:20,694 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.48 seconds 2025-02-14 08:55:20,694 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:55:20,694 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14090.58 MB 2025-02-14 08:55:20,694 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14660.35 MB 2025-02-14 08:55:20,694 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 569.77 MB 2025-02-14 08:55:20,694 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56126.08 MB 2025-02-14 08:55:20,694 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20285.75 MB 2025-02-14 08:55:20,694 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35840.33 MB 2025-02-14 08:55:20,694 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23561.95 MB 2025-02-14 08:55:20,713 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 08:55:20,713 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 08:55:20,713 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:55:20,713 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:55:20,713 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14660.35 MB 2025-02-14 08:55:20,713 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14936.40 MB 2025-02-14 08:55:20,713 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 276.05 MB 2025-02-14 08:55:20,713 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20285.75 MB 2025-02-14 08:55:20,713 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20285.75 MB 2025-02-14 08:55:20,713 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:55:20,713 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16925.37 MB 2025-02-14 08:55:21,509 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 08:55:21,509 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 08:55:21,509 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.79 seconds 2025-02-14 08:55:21,509 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:55:21,509 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14936.40 MB 2025-02-14 08:55:21,509 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15150.07 MB 2025-02-14 08:55:21,509 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 213.66 MB 2025-02-14 08:55:21,509 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20285.75 MB 2025-02-14 08:55:21,509 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20285.75 MB 2025-02-14 08:55:21,509 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:55:21,509 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19106.84 MB 2025-02-14 08:55:21,520 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 08:55:21,521 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 08:55:21,521 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:55:21,521 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:55:21,521 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15150.00 MB 2025-02-14 08:55:21,521 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15910.35 MB 2025-02-14 08:55:21,521 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 760.35 MB 2025-02-14 08:55:21,521 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20285.75 MB 2025-02-14 08:55:21,521 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20285.75 MB 2025-02-14 08:55:21,521 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:55:21,521 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16480.87 MB 2025-02-14 08:55:21,636 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 08:55:21,636 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 08:55:21,636 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 08:55:21,636 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:55:21,636 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15910.35 MB 2025-02-14 08:55:21,636 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16812.74 MB 2025-02-14 08:55:21,637 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 902.39 MB 2025-02-14 08:55:21,637 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20285.75 MB 2025-02-14 08:55:21,637 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20285.75 MB 2025-02-14 08:55:21,637 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:55:21,637 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19044.27 MB 2025-02-14 08:55:21,638 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 08:55:21,638 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 08:55:21,638 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 08:55:21,638 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:55:21,638 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15150.00 MB 2025-02-14 08:55:21,638 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16812.74 MB 2025-02-14 08:55:21,638 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1662.74 MB 2025-02-14 08:55:21,638 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20285.75 MB 2025-02-14 08:55:21,638 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20285.75 MB 2025-02-14 08:55:21,638 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:55:21,638 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19044.27 MB 2025-02-14 08:55:21,752 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 08:55:21,752 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 08:55:21,752 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 08:55:21,752 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:55:21,752 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17429.99 MB 2025-02-14 08:55:21,752 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17738.71 MB 2025-02-14 08:55:21,752 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 308.72 MB 2025-02-14 08:55:21,752 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20285.75 MB 2025-02-14 08:55:21,752 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20449.33 MB 2025-02-14 08:55:21,752 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 163.58 MB 2025-02-14 08:55:21,752 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18031.12 MB 2025-02-14 08:55:21,768 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 08:55:21,768 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 08:55:21,768 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:55:21,768 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:55:21,768 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17904.90 MB 2025-02-14 08:55:21,768 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18133.37 MB 2025-02-14 08:55:21,768 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.46 MB 2025-02-14 08:55:21,768 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20449.33 MB 2025-02-14 08:55:21,768 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20449.33 MB 2025-02-14 08:55:21,768 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:55:21,768 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18149.60 MB 2025-02-14 08:55:21,770 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 08:55:21,770 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 08:55:21,770 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.56 seconds 2025-02-14 08:55:21,770 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:55:21,770 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13529.64 MB 2025-02-14 08:55:21,770 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18334.17 MB 2025-02-14 08:55:21,770 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4804.53 MB 2025-02-14 08:55:21,770 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56126.08 MB 2025-02-14 08:55:21,770 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20449.33 MB 2025-02-14 08:55:21,770 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35676.75 MB 2025-02-14 08:55:21,770 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18334.17 MB 2025-02-14 08:55:22,061 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 08:55:22,061 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 08:55:22,061 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-14 08:55:22,061 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:55:22,061 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18334.17 MB 2025-02-14 08:55:22,061 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17401.93 MB 2025-02-14 08:55:22,061 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -932.24 MB 2025-02-14 08:55:22,061 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20449.33 MB 2025-02-14 08:55:22,061 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20449.33 MB 2025-02-14 08:55:22,061 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:55:22,062 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19136.82 MB 2025-02-14 08:55:22,086 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8151, cut from 8153 2025-02-14 08:55:22,086 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 08:55:22,101 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 08:55:22,101 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 08:55:22,101 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-14 08:55:22,101 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:55:22,101 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17401.93 MB 2025-02-14 08:55:22,101 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25829.27 MB 2025-02-14 08:55:22,101 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8427.34 MB 2025-02-14 08:55:22,101 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20449.33 MB 2025-02-14 08:55:22,101 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30924.60 MB 2025-02-14 08:55:22,101 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10475.27 MB 2025-02-14 08:55:22,101 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25829.27 MB 2025-02-14 08:55:22,336 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7943] 2025-02-14 08:55:22,338 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:55:22,338 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 08:55:22,340 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:55:22,340 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 08:55:22,347 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 08:55:22,349 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:55:22,349 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 08:55:22,350 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 08:55:32,222 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:55:32,222 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 08:55:32,229 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 08:55:32,235 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:55:32,235 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2100, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 08:55:32,237 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:55:32,237 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2100, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 08:56:04,645 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 08:56:04,645 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 08:56:04,645 - resource_logging.py:150 - __exit__ - DEBUG - Time: 32.40 seconds 2025-02-14 08:56:04,645 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:56:04,645 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27601.84 MB 2025-02-14 08:56:04,645 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35034.15 MB 2025-02-14 08:56:04,645 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7432.31 MB 2025-02-14 08:56:04,645 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39304.82 MB 2025-02-14 08:56:04,645 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41102.08 MB 2025-02-14 08:56:04,645 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1797.26 MB 2025-02-14 08:56:04,645 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43868.79 MB 2025-02-14 08:56:04,795 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 08:56:04,796 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 08:56:04,796 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 08:56:04,796 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:56:04,796 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35034.15 MB 2025-02-14 08:56:04,796 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26695.10 MB 2025-02-14 08:56:04,796 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8339.05 MB 2025-02-14 08:56:04,796 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41102.08 MB 2025-02-14 08:56:04,796 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65038.97 MB 2025-02-14 08:56:04,796 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 23936.89 MB 2025-02-14 08:56:04,796 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 55129.88 MB 2025-02-14 08:56:06,752 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 08:56:06,752 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 08:56:06,752 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.95 seconds 2025-02-14 08:56:06,752 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:56:06,752 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26695.10 MB 2025-02-14 08:56:06,752 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27225.94 MB 2025-02-14 08:56:06,752 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 08:56:06,753 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65038.97 MB 2025-02-14 08:56:06,753 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30895.24 MB 2025-02-14 08:56:06,753 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34143.73 MB 2025-02-14 08:56:06,753 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31206.31 MB 2025-02-14 08:56:06,766 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 08:56:06,766 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 08:56:06,766 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:56:06,766 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:56:06,766 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27225.94 MB 2025-02-14 08:56:06,766 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29115.47 MB 2025-02-14 08:56:06,766 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 08:56:06,766 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30895.24 MB 2025-02-14 08:56:06,766 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33726.40 MB 2025-02-14 08:56:06,766 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-14 08:56:06,766 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30532.90 MB 2025-02-14 08:56:06,979 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 08:56:06,980 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 08:56:06,980 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 08:56:06,980 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:56:06,980 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29115.47 MB 2025-02-14 08:56:06,980 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31357.33 MB 2025-02-14 08:56:06,980 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 08:56:06,980 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33726.40 MB 2025-02-14 08:56:06,980 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39388.71 MB 2025-02-14 08:56:06,980 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 08:56:06,980 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36901.61 MB 2025-02-14 08:56:06,980 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 08:56:06,980 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 08:56:06,980 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 08:56:06,980 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:56:06,980 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27225.94 MB 2025-02-14 08:56:06,980 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31357.33 MB 2025-02-14 08:56:06,980 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 08:56:06,980 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30895.24 MB 2025-02-14 08:56:06,980 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39388.71 MB 2025-02-14 08:56:06,980 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8493.47 MB 2025-02-14 08:56:06,980 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36901.61 MB 2025-02-14 08:56:07,155 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 08:56:07,155 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 08:56:07,155 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 08:56:07,155 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:56:07,155 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32890.87 MB 2025-02-14 08:56:07,155 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33657.87 MB 2025-02-14 08:56:07,155 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 08:56:07,155 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39388.71 MB 2025-02-14 08:56:07,155 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39803.94 MB 2025-02-14 08:56:07,155 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 08:56:07,155 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34365.66 MB 2025-02-14 08:56:07,175 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 08:56:07,175 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 08:56:07,175 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:56:07,175 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:56:07,175 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34070.76 MB 2025-02-14 08:56:07,175 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34299.55 MB 2025-02-14 08:56:07,175 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.79 MB 2025-02-14 08:56:07,175 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39803.94 MB 2025-02-14 08:56:07,175 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39803.94 MB 2025-02-14 08:56:07,175 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:56:07,175 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34513.64 MB 2025-02-14 08:56:07,176 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 08:56:07,176 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 08:56:07,176 - resource_logging.py:150 - __exit__ - DEBUG - Time: 34.94 seconds 2025-02-14 08:56:07,176 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:56:07,176 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20285.27 MB 2025-02-14 08:56:07,176 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34500.25 MB 2025-02-14 08:56:07,176 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14214.98 MB 2025-02-14 08:56:07,176 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39304.82 MB 2025-02-14 08:56:07,176 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39803.94 MB 2025-02-14 08:56:07,176 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 499.12 MB 2025-02-14 08:56:07,176 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34513.64 MB 2025-02-14 08:56:07,445 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 08:56:07,445 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 08:56:07,445 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 08:56:07,445 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:56:07,445 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34500.25 MB 2025-02-14 08:56:07,445 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25283.95 MB 2025-02-14 08:56:07,445 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9216.30 MB 2025-02-14 08:56:07,445 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39803.94 MB 2025-02-14 08:56:07,445 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39803.94 MB 2025-02-14 08:56:07,445 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:56:07,445 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37007.31 MB 2025-02-14 08:56:07,463 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8147, cut from 8149 2025-02-14 08:56:07,463 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-14 08:56:07,469 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 08:56:07,469 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 08:56:07,469 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:56:07,469 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:56:07,469 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25283.95 MB 2025-02-14 08:56:07,469 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33707.16 MB 2025-02-14 08:56:07,469 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8423.21 MB 2025-02-14 08:56:07,469 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39803.94 MB 2025-02-14 08:56:07,469 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48179.97 MB 2025-02-14 08:56:07,469 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8376.03 MB 2025-02-14 08:56:07,469 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33707.16 MB 2025-02-14 08:56:07,638 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7939] 2025-02-14 08:56:07,639 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:56:07,639 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 08:56:07,640 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:56:07,640 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 08:56:07,645 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 08:56:07,646 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:56:07,646 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 08:56:07,646 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-14 08:56:42,982 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:56:42,982 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 08:56:42,987 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 08:56:42,991 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:56:42,991 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 161, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 08:56:42,992 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:56:42,992 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 161, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 08:56:45,484 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 08:56:45,484 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 08:56:45,484 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.49 seconds 2025-02-14 08:56:45,484 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:56:45,484 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14090.58 MB 2025-02-14 08:56:45,484 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14660.35 MB 2025-02-14 08:56:45,484 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 569.77 MB 2025-02-14 08:56:45,484 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56556.00 MB 2025-02-14 08:56:45,484 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 16911.43 MB 2025-02-14 08:56:45,484 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -39644.56 MB 2025-02-14 08:56:45,484 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23562.76 MB 2025-02-14 08:56:45,496 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 08:56:45,496 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 08:56:45,496 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:56:45,496 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:56:45,496 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14660.35 MB 2025-02-14 08:56:45,496 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14937.06 MB 2025-02-14 08:56:45,496 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 276.71 MB 2025-02-14 08:56:45,496 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 16911.43 MB 2025-02-14 08:56:45,496 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18052.28 MB 2025-02-14 08:56:45,496 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1140.85 MB 2025-02-14 08:56:45,496 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16926.75 MB 2025-02-14 08:56:46,268 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 08:56:46,268 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 08:56:46,268 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.77 seconds 2025-02-14 08:56:46,268 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:56:46,268 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14937.06 MB 2025-02-14 08:56:46,268 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15150.72 MB 2025-02-14 08:56:46,268 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 213.66 MB 2025-02-14 08:56:46,268 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18052.28 MB 2025-02-14 08:56:46,268 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17481.86 MB 2025-02-14 08:56:46,268 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -570.43 MB 2025-02-14 08:56:46,268 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19108.53 MB 2025-02-14 08:56:46,276 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 08:56:46,276 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 08:56:46,276 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 08:56:46,276 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:56:46,276 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15150.66 MB 2025-02-14 08:56:46,276 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15911.01 MB 2025-02-14 08:56:46,276 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 760.35 MB 2025-02-14 08:56:46,276 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17481.86 MB 2025-02-14 08:56:46,276 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17863.54 MB 2025-02-14 08:56:46,276 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 381.68 MB 2025-02-14 08:56:46,276 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16481.53 MB 2025-02-14 08:56:46,365 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 08:56:46,365 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 08:56:46,365 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 08:56:46,365 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:56:46,365 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15911.01 MB 2025-02-14 08:56:46,365 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16813.39 MB 2025-02-14 08:56:46,365 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 902.39 MB 2025-02-14 08:56:46,365 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17863.54 MB 2025-02-14 08:56:46,365 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20153.63 MB 2025-02-14 08:56:46,365 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2290.09 MB 2025-02-14 08:56:46,365 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19046.76 MB 2025-02-14 08:56:46,366 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 08:56:46,366 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 08:56:46,366 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 08:56:46,366 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:56:46,366 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15150.66 MB 2025-02-14 08:56:46,366 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16813.39 MB 2025-02-14 08:56:46,366 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1662.74 MB 2025-02-14 08:56:46,366 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17481.86 MB 2025-02-14 08:56:46,366 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20153.63 MB 2025-02-14 08:56:46,366 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2671.77 MB 2025-02-14 08:56:46,366 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19046.76 MB 2025-02-14 08:56:46,434 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 08:56:46,434 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 08:56:46,434 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 08:56:46,434 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:56:46,434 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17430.65 MB 2025-02-14 08:56:46,434 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17741.20 MB 2025-02-14 08:56:46,434 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 310.55 MB 2025-02-14 08:56:46,434 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20153.63 MB 2025-02-14 08:56:46,434 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20315.11 MB 2025-02-14 08:56:46,434 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 161.48 MB 2025-02-14 08:56:46,434 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18034.51 MB 2025-02-14 08:56:46,444 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 08:56:46,444 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 08:56:46,444 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:56:46,444 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:56:46,444 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17907.39 MB 2025-02-14 08:56:46,444 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18135.97 MB 2025-02-14 08:56:46,444 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.57 MB 2025-02-14 08:56:46,444 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20315.11 MB 2025-02-14 08:56:46,444 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20315.11 MB 2025-02-14 08:56:46,444 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:56:46,444 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18151.30 MB 2025-02-14 08:56:46,445 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 08:56:46,445 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 08:56:46,445 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.45 seconds 2025-02-14 08:56:46,445 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:56:46,445 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13529.64 MB 2025-02-14 08:56:46,445 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18336.72 MB 2025-02-14 08:56:46,445 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4807.08 MB 2025-02-14 08:56:46,445 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56556.00 MB 2025-02-14 08:56:46,445 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20315.11 MB 2025-02-14 08:56:46,445 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -36240.88 MB 2025-02-14 08:56:46,445 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18336.72 MB 2025-02-14 08:56:46,713 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 08:56:46,713 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 08:56:46,713 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 08:56:46,713 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:56:46,713 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18336.72 MB 2025-02-14 08:56:46,713 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17402.74 MB 2025-02-14 08:56:46,713 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -933.98 MB 2025-02-14 08:56:46,713 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20315.11 MB 2025-02-14 08:56:46,713 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20315.11 MB 2025-02-14 08:56:46,713 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:56:46,713 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19139.18 MB 2025-02-14 08:56:46,731 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8149, cut from 8151 2025-02-14 08:56:46,731 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 08:56:46,737 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 08:56:46,737 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 08:56:46,737 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:56:46,737 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:56:46,737 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17402.74 MB 2025-02-14 08:56:46,737 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25828.92 MB 2025-02-14 08:56:46,737 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8426.18 MB 2025-02-14 08:56:46,737 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20315.11 MB 2025-02-14 08:56:46,737 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30786.19 MB 2025-02-14 08:56:46,737 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10471.08 MB 2025-02-14 08:56:46,737 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25828.92 MB 2025-02-14 08:56:46,900 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7941] 2025-02-14 08:56:46,901 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:56:46,901 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 08:56:46,902 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:56:46,902 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 08:56:46,907 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 08:56:46,908 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:56:46,908 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 08:56:46,908 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 08:57:35,648 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:57:35,648 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 08:57:35,653 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 08:57:35,656 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:57:35,656 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 650, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 08:57:35,657 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:57:35,657 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 650, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 08:57:45,581 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 08:57:45,581 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 08:57:45,582 - resource_logging.py:150 - __exit__ - DEBUG - Time: 9.92 seconds 2025-02-14 08:57:45,582 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:57:45,582 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17498.01 MB 2025-02-14 08:57:45,582 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19798.59 MB 2025-02-14 08:57:45,582 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2300.58 MB 2025-02-14 08:57:45,582 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39162.22 MB 2025-02-14 08:57:45,582 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25495.08 MB 2025-02-14 08:57:45,582 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13667.14 MB 2025-02-14 08:57:45,582 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28782.11 MB 2025-02-14 08:57:45,641 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 08:57:45,641 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 08:57:45,641 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 08:57:45,641 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:57:45,641 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19798.59 MB 2025-02-14 08:57:45,641 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19157.00 MB 2025-02-14 08:57:45,641 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -641.59 MB 2025-02-14 08:57:45,641 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25495.08 MB 2025-02-14 08:57:45,641 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31421.63 MB 2025-02-14 08:57:45,641 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5926.55 MB 2025-02-14 08:57:45,641 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28196.38 MB 2025-02-14 08:57:47,541 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 08:57:47,542 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 08:57:47,542 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.90 seconds 2025-02-14 08:57:47,542 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:57:47,542 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19157.00 MB 2025-02-14 08:57:47,542 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19687.84 MB 2025-02-14 08:57:47,542 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 08:57:47,542 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31421.63 MB 2025-02-14 08:57:47,542 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24610.08 MB 2025-02-14 08:57:47,542 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6811.55 MB 2025-02-14 08:57:47,542 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23667.17 MB 2025-02-14 08:57:47,556 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 08:57:47,556 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 08:57:47,556 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:57:47,556 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:57:47,556 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19687.84 MB 2025-02-14 08:57:47,556 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21577.37 MB 2025-02-14 08:57:47,556 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 08:57:47,556 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24610.08 MB 2025-02-14 08:57:47,556 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25553.80 MB 2025-02-14 08:57:47,556 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 08:57:47,556 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22994.80 MB 2025-02-14 08:57:47,767 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 08:57:47,767 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 08:57:47,767 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 08:57:47,767 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:57:47,767 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21577.37 MB 2025-02-14 08:57:47,767 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23819.23 MB 2025-02-14 08:57:47,767 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 08:57:47,767 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25553.80 MB 2025-02-14 08:57:47,767 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31687.97 MB 2025-02-14 08:57:47,767 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 08:57:47,767 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29363.51 MB 2025-02-14 08:57:47,768 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 08:57:47,768 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 08:57:47,768 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 08:57:47,768 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:57:47,768 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19687.84 MB 2025-02-14 08:57:47,768 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23819.23 MB 2025-02-14 08:57:47,768 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 08:57:47,768 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24610.08 MB 2025-02-14 08:57:47,768 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31687.97 MB 2025-02-14 08:57:47,768 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7077.89 MB 2025-02-14 08:57:47,768 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29363.51 MB 2025-02-14 08:57:47,939 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 08:57:47,939 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 08:57:47,939 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 08:57:47,939 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:57:47,939 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25352.77 MB 2025-02-14 08:57:47,939 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26119.77 MB 2025-02-14 08:57:47,939 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 08:57:47,939 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31687.97 MB 2025-02-14 08:57:47,939 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32101.11 MB 2025-02-14 08:57:47,939 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 08:57:47,939 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26827.56 MB 2025-02-14 08:57:47,958 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 08:57:47,958 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 08:57:47,958 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:57:47,958 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:57:47,958 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26532.66 MB 2025-02-14 08:57:47,958 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26764.18 MB 2025-02-14 08:57:47,958 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 231.52 MB 2025-02-14 08:57:47,958 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32101.11 MB 2025-02-14 08:57:47,958 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32101.11 MB 2025-02-14 08:57:47,958 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:57:47,958 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26977.46 MB 2025-02-14 08:57:47,959 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 08:57:47,959 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 08:57:47,959 - resource_logging.py:150 - __exit__ - DEBUG - Time: 12.30 seconds 2025-02-14 08:57:47,959 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:57:47,959 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15233.36 MB 2025-02-14 08:57:47,959 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26965.25 MB 2025-02-14 08:57:47,959 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11731.89 MB 2025-02-14 08:57:47,959 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39162.22 MB 2025-02-14 08:57:47,959 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32101.11 MB 2025-02-14 08:57:47,959 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7061.11 MB 2025-02-14 08:57:47,959 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26977.46 MB 2025-02-14 08:57:48,225 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 08:57:48,225 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 08:57:48,225 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 08:57:48,225 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:57:48,225 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26965.25 MB 2025-02-14 08:57:48,225 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20237.75 MB 2025-02-14 08:57:48,225 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6727.50 MB 2025-02-14 08:57:48,225 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32101.11 MB 2025-02-14 08:57:48,225 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32101.11 MB 2025-02-14 08:57:48,225 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:57:48,225 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29476.92 MB 2025-02-14 08:57:48,243 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 08:57:48,243 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 08:57:48,249 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 08:57:48,249 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 08:57:48,249 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:57:48,249 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:57:48,250 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20237.75 MB 2025-02-14 08:57:48,250 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28676.77 MB 2025-02-14 08:57:48,250 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 08:57:48,250 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32101.11 MB 2025-02-14 08:57:48,250 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40491.81 MB 2025-02-14 08:57:48,250 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 08:57:48,250 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28676.77 MB 2025-02-14 08:57:48,413 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 08:57:48,415 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:57:48,415 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 08:57:48,416 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:57:48,416 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 08:57:48,421 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 08:57:48,422 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:57:48,422 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 08:57:48,422 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 08:58:43,765 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:58:43,765 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 08:58:43,771 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 08:58:43,777 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:58:43,777 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1357, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 08:58:43,779 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:58:43,779 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1357, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 08:59:04,586 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 08:59:04,586 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 08:59:04,586 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.80 seconds 2025-02-14 08:59:04,586 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:59:04,586 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22424.50 MB 2025-02-14 08:59:04,586 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27226.98 MB 2025-02-14 08:59:04,586 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4802.48 MB 2025-02-14 08:59:04,586 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53076.82 MB 2025-02-14 08:59:04,586 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38482.74 MB 2025-02-14 08:59:04,586 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14594.08 MB 2025-02-14 08:59:04,586 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36199.23 MB 2025-02-14 08:59:04,633 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 08:59:04,634 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 08:59:04,634 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-14 08:59:04,634 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:59:04,634 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27226.98 MB 2025-02-14 08:59:04,634 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21849.25 MB 2025-02-14 08:59:04,634 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5377.73 MB 2025-02-14 08:59:04,634 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38482.74 MB 2025-02-14 08:59:04,634 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38482.74 MB 2025-02-14 08:59:04,634 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:59:04,634 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30924.96 MB 2025-02-14 08:59:05,887 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 08:59:05,888 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 08:59:05,888 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.25 seconds 2025-02-14 08:59:05,888 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:59:05,888 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21849.25 MB 2025-02-14 08:59:05,888 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22194.29 MB 2025-02-14 08:59:05,888 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 345.05 MB 2025-02-14 08:59:05,888 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38482.74 MB 2025-02-14 08:59:05,888 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29485.96 MB 2025-02-14 08:59:05,888 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8996.78 MB 2025-02-14 08:59:05,888 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26189.55 MB 2025-02-14 08:59:05,897 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 08:59:05,897 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 08:59:05,897 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:59:05,897 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:59:05,897 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22194.29 MB 2025-02-14 08:59:05,897 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23422.27 MB 2025-02-14 08:59:05,897 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1227.97 MB 2025-02-14 08:59:05,897 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29485.96 MB 2025-02-14 08:59:05,897 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29485.96 MB 2025-02-14 08:59:05,897 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:59:05,897 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24343.60 MB 2025-02-14 08:59:06,034 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 08:59:06,034 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 08:59:06,034 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-14 08:59:06,034 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:59:06,034 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23422.27 MB 2025-02-14 08:59:06,034 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24879.50 MB 2025-02-14 08:59:06,034 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1457.23 MB 2025-02-14 08:59:06,034 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29485.96 MB 2025-02-14 08:59:06,034 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29485.96 MB 2025-02-14 08:59:06,034 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:59:06,034 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28483.26 MB 2025-02-14 08:59:06,035 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 08:59:06,035 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 08:59:06,035 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 08:59:06,035 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:59:06,035 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22194.29 MB 2025-02-14 08:59:06,035 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24879.50 MB 2025-02-14 08:59:06,035 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2685.20 MB 2025-02-14 08:59:06,035 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29485.96 MB 2025-02-14 08:59:06,035 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29485.96 MB 2025-02-14 08:59:06,035 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:59:06,035 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28483.26 MB 2025-02-14 08:59:06,146 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 08:59:06,146 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 08:59:06,146 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 08:59:06,147 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:59:06,147 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25876.30 MB 2025-02-14 08:59:06,147 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26374.85 MB 2025-02-14 08:59:06,147 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 498.55 MB 2025-02-14 08:59:06,147 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29485.96 MB 2025-02-14 08:59:06,147 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29754.39 MB 2025-02-14 08:59:06,147 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 268.44 MB 2025-02-14 08:59:06,147 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26834.91 MB 2025-02-14 08:59:06,160 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 08:59:06,160 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 08:59:06,160 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:59:06,160 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:59:06,160 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26643.23 MB 2025-02-14 08:59:06,160 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26856.50 MB 2025-02-14 08:59:06,160 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 213.27 MB 2025-02-14 08:59:06,160 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29754.39 MB 2025-02-14 08:59:06,160 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29754.39 MB 2025-02-14 08:59:06,160 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:59:06,160 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26933.48 MB 2025-02-14 08:59:06,161 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 08:59:06,161 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 08:59:06,161 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.38 seconds 2025-02-14 08:59:06,161 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:59:06,161 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17696.60 MB 2025-02-14 08:59:06,161 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27057.57 MB 2025-02-14 08:59:06,161 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 9360.97 MB 2025-02-14 08:59:06,161 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53076.82 MB 2025-02-14 08:59:06,161 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29754.39 MB 2025-02-14 08:59:06,161 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23322.43 MB 2025-02-14 08:59:06,161 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27057.57 MB 2025-02-14 08:59:06,430 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 08:59:06,431 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 08:59:06,431 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 08:59:06,431 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:59:06,431 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27057.57 MB 2025-02-14 08:59:06,431 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30071.61 MB 2025-02-14 08:59:06,431 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-14 08:59:06,431 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29754.39 MB 2025-02-14 08:59:06,431 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31499.22 MB 2025-02-14 08:59:06,431 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1744.83 MB 2025-02-14 08:59:06,431 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30373.24 MB 2025-02-14 08:59:06,448 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 08:59:06,449 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 08:59:06,455 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 08:59:06,455 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 08:59:06,455 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:59:06,455 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:59:06,455 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22040.29 MB 2025-02-14 08:59:06,455 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30479.31 MB 2025-02-14 08:59:06,455 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 08:59:06,455 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31499.22 MB 2025-02-14 08:59:06,455 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39889.93 MB 2025-02-14 08:59:06,455 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 08:59:06,455 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30479.31 MB 2025-02-14 08:59:06,618 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 08:59:06,619 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:59:06,619 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 08:59:06,620 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:59:06,620 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 08:59:06,625 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 08:59:06,626 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:59:06,626 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 08:59:06,626 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 08:59:30,384 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:59:30,384 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 08:59:30,389 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 08:59:30,392 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:59:30,392 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1188, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 08:59:30,393 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:59:30,393 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1188, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 08:59:48,671 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 08:59:48,672 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 08:59:48,672 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.27 seconds 2025-02-14 08:59:48,672 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:59:48,672 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21246.88 MB 2025-02-14 08:59:48,672 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25451.67 MB 2025-02-14 08:59:48,672 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4204.79 MB 2025-02-14 08:59:48,672 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52474.94 MB 2025-02-14 08:59:48,672 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29502.73 MB 2025-02-14 08:59:48,672 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22972.20 MB 2025-02-14 08:59:48,672 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34343.29 MB 2025-02-14 08:59:48,752 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 08:59:48,752 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 08:59:48,752 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 08:59:48,752 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:59:48,752 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25451.67 MB 2025-02-14 08:59:48,752 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21953.89 MB 2025-02-14 08:59:48,752 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3497.78 MB 2025-02-14 08:59:48,752 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29502.73 MB 2025-02-14 08:59:48,752 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45306.87 MB 2025-02-14 08:59:48,752 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 15804.14 MB 2025-02-14 08:59:48,752 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38021.94 MB 2025-02-14 08:59:50,674 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 08:59:50,674 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 08:59:50,674 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 08:59:50,674 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:59:50,674 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21953.89 MB 2025-02-14 08:59:50,674 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22484.73 MB 2025-02-14 08:59:50,674 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 08:59:50,674 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45306.87 MB 2025-02-14 08:59:50,674 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26713.52 MB 2025-02-14 08:59:50,674 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18593.35 MB 2025-02-14 08:59:50,674 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26465.11 MB 2025-02-14 08:59:50,688 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 08:59:50,688 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 08:59:50,688 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 08:59:50,688 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:59:50,688 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22484.73 MB 2025-02-14 08:59:50,688 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24374.27 MB 2025-02-14 08:59:50,688 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 08:59:50,688 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26713.52 MB 2025-02-14 08:59:50,688 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28600.96 MB 2025-02-14 08:59:50,688 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 08:59:50,688 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25791.70 MB 2025-02-14 08:59:50,897 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 08:59:50,897 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 08:59:50,897 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 08:59:50,897 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:59:50,897 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24374.27 MB 2025-02-14 08:59:50,897 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26616.12 MB 2025-02-14 08:59:50,897 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 08:59:50,897 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28600.96 MB 2025-02-14 08:59:50,897 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34263.27 MB 2025-02-14 08:59:50,897 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 08:59:50,897 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32160.41 MB 2025-02-14 08:59:50,898 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 08:59:50,898 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 08:59:50,898 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 08:59:50,898 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:59:50,898 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22484.73 MB 2025-02-14 08:59:50,898 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26616.12 MB 2025-02-14 08:59:50,898 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 08:59:50,898 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26713.52 MB 2025-02-14 08:59:50,898 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34263.27 MB 2025-02-14 08:59:50,898 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-14 08:59:50,898 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32160.41 MB 2025-02-14 08:59:51,064 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 08:59:51,064 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 08:59:51,064 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 08:59:51,064 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:59:51,064 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28149.67 MB 2025-02-14 08:59:51,064 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28916.67 MB 2025-02-14 08:59:51,064 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 08:59:51,064 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34263.27 MB 2025-02-14 08:59:51,064 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34678.51 MB 2025-02-14 08:59:51,064 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 08:59:51,064 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29624.46 MB 2025-02-14 08:59:51,083 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 08:59:51,083 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 08:59:51,083 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:59:51,083 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:59:51,083 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29329.56 MB 2025-02-14 08:59:51,083 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29558.03 MB 2025-02-14 08:59:51,083 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.47 MB 2025-02-14 08:59:51,083 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34678.51 MB 2025-02-14 08:59:51,083 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34678.51 MB 2025-02-14 08:59:51,083 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:59:51,083 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29800.75 MB 2025-02-14 08:59:51,084 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 08:59:51,084 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 08:59:51,084 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.69 seconds 2025-02-14 08:59:51,084 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:59:51,084 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17107.79 MB 2025-02-14 08:59:51,084 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29758.24 MB 2025-02-14 08:59:51,084 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12650.45 MB 2025-02-14 08:59:51,084 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52474.94 MB 2025-02-14 08:59:51,084 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34678.51 MB 2025-02-14 08:59:51,084 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17796.43 MB 2025-02-14 08:59:51,084 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29800.75 MB 2025-02-14 08:59:51,352 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 08:59:51,352 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 08:59:51,352 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 08:59:51,352 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:59:51,352 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29758.24 MB 2025-02-14 08:59:51,352 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22098.85 MB 2025-02-14 08:59:51,352 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7659.39 MB 2025-02-14 08:59:51,352 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34678.51 MB 2025-02-14 08:59:51,352 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34678.51 MB 2025-02-14 08:59:51,352 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 08:59:51,352 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32259.15 MB 2025-02-14 08:59:51,369 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8127, cut from 8129 2025-02-14 08:59:51,370 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 08:59:51,376 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 08:59:51,376 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 08:59:51,376 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 08:59:51,376 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 08:59:51,376 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22098.85 MB 2025-02-14 08:59:51,376 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30502.41 MB 2025-02-14 08:59:51,376 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8403.56 MB 2025-02-14 08:59:51,376 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34678.51 MB 2025-02-14 08:59:51,376 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43033.56 MB 2025-02-14 08:59:51,376 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8355.05 MB 2025-02-14 08:59:51,376 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30502.41 MB 2025-02-14 08:59:51,539 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7919] 2025-02-14 08:59:51,540 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:59:51,540 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 08:59:51,541 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:59:51,541 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 08:59:51,546 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 08:59:51,548 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 08:59:51,548 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 08:59:51,548 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 09:00:31,445 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:00:31,445 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 09:00:31,450 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 09:00:31,454 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:00:31,454 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 473, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 09:00:31,455 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:00:31,455 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 473, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 09:00:38,733 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 09:00:38,733 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 09:00:38,733 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.27 seconds 2025-02-14 09:00:38,733 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:00:38,733 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16264.65 MB 2025-02-14 09:00:38,733 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17938.57 MB 2025-02-14 09:00:38,734 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1673.92 MB 2025-02-14 09:00:38,734 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51388.61 MB 2025-02-14 09:00:38,734 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22359.83 MB 2025-02-14 09:00:38,734 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29028.78 MB 2025-02-14 09:00:38,734 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26869.29 MB 2025-02-14 09:00:38,767 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 09:00:38,767 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 09:00:38,767 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 09:00:38,767 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:00:38,767 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17938.57 MB 2025-02-14 09:00:38,767 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18237.88 MB 2025-02-14 09:00:38,767 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 299.31 MB 2025-02-14 09:00:38,767 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22359.83 MB 2025-02-14 09:00:38,767 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28328.33 MB 2025-02-14 09:00:38,767 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5968.49 MB 2025-02-14 09:00:38,767 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25442.25 MB 2025-02-14 09:00:40,673 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 09:00:40,673 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 09:00:40,674 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 09:00:40,674 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:00:40,674 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18237.88 MB 2025-02-14 09:00:40,674 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18768.72 MB 2025-02-14 09:00:40,674 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 09:00:40,674 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28328.33 MB 2025-02-14 09:00:40,674 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21864.91 MB 2025-02-14 09:00:40,674 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6463.42 MB 2025-02-14 09:00:40,674 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22749.09 MB 2025-02-14 09:00:40,687 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 09:00:40,687 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 09:00:40,687 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:00:40,687 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:00:40,687 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18768.72 MB 2025-02-14 09:00:40,687 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20658.25 MB 2025-02-14 09:00:40,687 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 09:00:40,687 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21864.91 MB 2025-02-14 09:00:40,687 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24696.06 MB 2025-02-14 09:00:40,687 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-14 09:00:40,687 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22075.68 MB 2025-02-14 09:00:40,901 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 09:00:40,901 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 09:00:40,901 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 09:00:40,901 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:00:40,901 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20658.25 MB 2025-02-14 09:00:40,901 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22900.11 MB 2025-02-14 09:00:40,902 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 09:00:40,902 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24696.06 MB 2025-02-14 09:00:40,902 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30830.23 MB 2025-02-14 09:00:40,902 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 09:00:40,902 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28444.39 MB 2025-02-14 09:00:40,902 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 09:00:40,902 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 09:00:40,902 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 09:00:40,902 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:00:40,902 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18768.72 MB 2025-02-14 09:00:40,902 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22900.11 MB 2025-02-14 09:00:40,902 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 09:00:40,902 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21864.91 MB 2025-02-14 09:00:40,902 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30830.23 MB 2025-02-14 09:00:40,902 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8965.32 MB 2025-02-14 09:00:40,902 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28444.39 MB 2025-02-14 09:00:41,069 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 09:00:41,069 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 09:00:41,069 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 09:00:41,069 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:00:41,069 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24433.65 MB 2025-02-14 09:00:41,069 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25200.65 MB 2025-02-14 09:00:41,069 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 09:00:41,069 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30830.23 MB 2025-02-14 09:00:41,069 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31245.47 MB 2025-02-14 09:00:41,069 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 09:00:41,069 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25908.44 MB 2025-02-14 09:00:41,088 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 09:00:41,088 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 09:00:41,088 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:00:41,088 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:00:41,088 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25613.54 MB 2025-02-14 09:00:41,088 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25842.70 MB 2025-02-14 09:00:41,088 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.16 MB 2025-02-14 09:00:41,088 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31245.47 MB 2025-02-14 09:00:41,088 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31245.47 MB 2025-02-14 09:00:41,088 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:00:41,088 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26028.18 MB 2025-02-14 09:00:41,089 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 09:00:41,089 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 09:00:41,089 - resource_logging.py:150 - __exit__ - DEBUG - Time: 9.63 seconds 2025-02-14 09:00:41,089 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:00:41,089 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14616.68 MB 2025-02-14 09:00:41,089 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26043.77 MB 2025-02-14 09:00:41,089 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11427.10 MB 2025-02-14 09:00:41,089 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51388.61 MB 2025-02-14 09:00:41,089 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31245.47 MB 2025-02-14 09:00:41,089 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20143.14 MB 2025-02-14 09:00:41,089 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26043.77 MB 2025-02-14 09:00:41,355 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 09:00:41,355 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 09:00:41,355 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 09:00:41,355 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:00:41,355 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26043.77 MB 2025-02-14 09:00:41,355 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19621.07 MB 2025-02-14 09:00:41,355 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6422.71 MB 2025-02-14 09:00:41,355 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31245.47 MB 2025-02-14 09:00:41,355 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31245.47 MB 2025-02-14 09:00:41,355 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:00:41,355 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28555.44 MB 2025-02-14 09:00:41,372 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 09:00:41,373 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 09:00:41,379 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 09:00:41,379 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 09:00:41,379 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:00:41,379 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:00:41,379 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19621.07 MB 2025-02-14 09:00:41,379 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28060.09 MB 2025-02-14 09:00:41,379 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 09:00:41,379 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31245.47 MB 2025-02-14 09:00:41,379 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41735.42 MB 2025-02-14 09:00:41,379 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 09:00:41,379 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28060.09 MB 2025-02-14 09:00:41,542 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 09:00:41,543 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:00:41,543 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 09:00:41,544 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:00:41,544 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 09:00:41,549 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 09:00:41,550 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:00:41,550 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 09:00:41,550 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 09:01:42,656 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:01:42,656 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 09:01:42,661 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 09:01:42,665 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:01:42,665 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1047, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 09:01:42,666 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:01:42,666 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1047, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 09:01:58,720 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 09:01:58,720 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 09:01:58,720 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.05 seconds 2025-02-14 09:01:58,720 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:01:58,720 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20264.37 MB 2025-02-14 09:01:58,720 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23970.04 MB 2025-02-14 09:01:58,720 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3705.67 MB 2025-02-14 09:01:58,720 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54320.43 MB 2025-02-14 09:01:58,720 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31102.86 MB 2025-02-14 09:01:58,720 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23217.57 MB 2025-02-14 09:01:58,720 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32906.63 MB 2025-02-14 09:01:58,773 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 09:01:58,773 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 09:01:58,773 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-14 09:01:58,773 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:01:58,773 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23970.04 MB 2025-02-14 09:01:58,773 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21220.88 MB 2025-02-14 09:01:58,773 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2749.16 MB 2025-02-14 09:01:58,773 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31102.86 MB 2025-02-14 09:01:58,773 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36022.78 MB 2025-02-14 09:01:58,773 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4919.92 MB 2025-02-14 09:01:58,773 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32363.97 MB 2025-02-14 09:02:00,678 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 09:02:00,678 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 09:02:00,678 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.90 seconds 2025-02-14 09:02:00,678 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:02:00,678 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21220.88 MB 2025-02-14 09:02:00,678 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21751.72 MB 2025-02-14 09:02:00,678 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 09:02:00,678 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36022.78 MB 2025-02-14 09:02:00,678 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28812.77 MB 2025-02-14 09:02:00,678 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7210.01 MB 2025-02-14 09:02:00,678 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25731.05 MB 2025-02-14 09:02:00,691 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 09:02:00,691 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 09:02:00,691 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:02:00,691 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:02:00,691 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21751.72 MB 2025-02-14 09:02:00,691 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23641.25 MB 2025-02-14 09:02:00,691 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 09:02:00,691 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28812.77 MB 2025-02-14 09:02:00,691 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28812.77 MB 2025-02-14 09:02:00,691 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:02:00,691 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25058.68 MB 2025-02-14 09:02:00,902 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 09:02:00,902 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 09:02:00,902 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 09:02:00,902 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:02:00,903 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23641.25 MB 2025-02-14 09:02:00,903 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25883.11 MB 2025-02-14 09:02:00,903 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 09:02:00,903 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28812.77 MB 2025-02-14 09:02:00,903 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34003.22 MB 2025-02-14 09:02:00,903 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 09:02:00,903 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31427.39 MB 2025-02-14 09:02:00,903 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 09:02:00,903 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 09:02:00,903 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 09:02:00,903 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:02:00,903 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21751.72 MB 2025-02-14 09:02:00,903 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25883.11 MB 2025-02-14 09:02:00,903 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 09:02:00,903 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28812.77 MB 2025-02-14 09:02:00,903 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34003.22 MB 2025-02-14 09:02:00,903 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 09:02:00,903 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31427.39 MB 2025-02-14 09:02:01,070 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 09:02:01,070 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 09:02:01,070 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 09:02:01,071 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:02:01,071 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27416.65 MB 2025-02-14 09:02:01,071 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28183.65 MB 2025-02-14 09:02:01,071 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 09:02:01,071 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34003.22 MB 2025-02-14 09:02:01,071 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34418.46 MB 2025-02-14 09:02:01,071 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 09:02:01,071 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28891.44 MB 2025-02-14 09:02:01,089 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 09:02:01,089 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 09:02:01,089 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:02:01,089 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:02:01,089 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28596.54 MB 2025-02-14 09:02:01,089 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28828.73 MB 2025-02-14 09:02:01,089 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 232.19 MB 2025-02-14 09:02:01,089 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34418.46 MB 2025-02-14 09:02:01,089 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34418.46 MB 2025-02-14 09:02:01,089 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:02:01,089 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29017.48 MB 2025-02-14 09:02:01,090 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 09:02:01,090 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 09:02:01,090 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.42 seconds 2025-02-14 09:02:01,090 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:02:01,090 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16616.54 MB 2025-02-14 09:02:01,091 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29029.81 MB 2025-02-14 09:02:01,091 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12413.27 MB 2025-02-14 09:02:01,091 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54320.43 MB 2025-02-14 09:02:01,091 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34418.46 MB 2025-02-14 09:02:01,091 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19901.97 MB 2025-02-14 09:02:01,091 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29029.81 MB 2025-02-14 09:02:01,356 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 09:02:01,356 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 09:02:01,356 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 09:02:01,356 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:02:01,356 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29029.81 MB 2025-02-14 09:02:01,356 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21620.93 MB 2025-02-14 09:02:01,356 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7408.88 MB 2025-02-14 09:02:01,357 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34418.46 MB 2025-02-14 09:02:01,357 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34418.46 MB 2025-02-14 09:02:01,357 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:02:01,357 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31541.47 MB 2025-02-14 09:02:01,374 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 09:02:01,374 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 09:02:01,381 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 09:02:01,381 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 09:02:01,381 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:02:01,381 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:02:01,381 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21620.93 MB 2025-02-14 09:02:01,381 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30059.95 MB 2025-02-14 09:02:01,381 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 09:02:01,381 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34418.46 MB 2025-02-14 09:02:01,381 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42809.16 MB 2025-02-14 09:02:01,381 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 09:02:01,381 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30059.95 MB 2025-02-14 09:02:01,545 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 09:02:01,547 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:02:01,547 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 09:02:01,548 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:02:01,548 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 09:02:01,553 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 09:02:01,554 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:02:01,554 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 09:02:01,554 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 09:02:50,155 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:02:50,155 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 09:02:50,162 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 09:02:50,167 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:02:50,167 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1593, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 09:02:50,169 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:02:50,169 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1593, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 09:03:14,691 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 09:03:14,691 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 09:03:14,691 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.51 seconds 2025-02-14 09:03:14,691 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:03:14,691 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24068.99 MB 2025-02-14 09:03:14,691 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29706.52 MB 2025-02-14 09:03:14,691 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5637.54 MB 2025-02-14 09:03:14,691 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55394.17 MB 2025-02-14 09:03:14,691 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39325.79 MB 2025-02-14 09:03:14,691 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16068.38 MB 2025-02-14 09:03:14,691 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38524.00 MB 2025-02-14 09:03:14,774 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 09:03:14,774 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 09:03:14,774 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 09:03:14,774 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:03:14,774 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29706.52 MB 2025-02-14 09:03:14,774 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24059.36 MB 2025-02-14 09:03:14,774 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5647.16 MB 2025-02-14 09:03:14,774 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39325.79 MB 2025-02-14 09:03:14,774 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48926.56 MB 2025-02-14 09:03:14,774 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9600.76 MB 2025-02-14 09:03:14,774 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44061.19 MB 2025-02-14 09:03:16,681 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 09:03:16,681 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 09:03:16,681 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 09:03:16,681 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:03:16,681 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24059.36 MB 2025-02-14 09:03:16,681 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24590.20 MB 2025-02-14 09:03:16,681 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 09:03:16,681 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48926.56 MB 2025-02-14 09:03:16,681 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29492.25 MB 2025-02-14 09:03:16,681 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19434.31 MB 2025-02-14 09:03:16,681 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28569.54 MB 2025-02-14 09:03:16,697 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 09:03:16,697 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 09:03:16,697 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:03:16,697 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:03:16,697 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24590.20 MB 2025-02-14 09:03:16,697 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26479.74 MB 2025-02-14 09:03:16,697 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 09:03:16,697 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29492.25 MB 2025-02-14 09:03:16,697 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30435.97 MB 2025-02-14 09:03:16,697 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 09:03:16,697 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27897.17 MB 2025-02-14 09:03:16,911 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 09:03:16,911 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 09:03:16,911 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 09:03:16,911 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:03:16,911 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26479.74 MB 2025-02-14 09:03:16,912 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28721.59 MB 2025-02-14 09:03:16,912 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 09:03:16,912 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30435.97 MB 2025-02-14 09:03:16,912 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36570.14 MB 2025-02-14 09:03:16,912 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 09:03:16,912 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34265.87 MB 2025-02-14 09:03:16,912 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 09:03:16,912 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 09:03:16,912 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 09:03:16,912 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:03:16,912 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24590.20 MB 2025-02-14 09:03:16,912 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28721.59 MB 2025-02-14 09:03:16,912 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 09:03:16,912 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29492.25 MB 2025-02-14 09:03:16,912 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36570.14 MB 2025-02-14 09:03:16,912 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7077.89 MB 2025-02-14 09:03:16,912 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34265.87 MB 2025-02-14 09:03:17,141 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 09:03:17,141 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 09:03:17,141 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 09:03:17,141 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:03:17,141 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30255.14 MB 2025-02-14 09:03:17,141 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31022.14 MB 2025-02-14 09:03:17,141 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 09:03:17,141 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36570.14 MB 2025-02-14 09:03:17,141 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36985.37 MB 2025-02-14 09:03:17,141 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 09:03:17,141 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31729.93 MB 2025-02-14 09:03:17,168 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 09:03:17,168 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 09:03:17,168 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:03:17,168 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:03:17,168 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31435.03 MB 2025-02-14 09:03:17,168 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31662.98 MB 2025-02-14 09:03:17,168 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.95 MB 2025-02-14 09:03:17,168 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36985.37 MB 2025-02-14 09:03:17,168 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36985.37 MB 2025-02-14 09:03:17,168 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:03:17,168 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31898.72 MB 2025-02-14 09:03:17,170 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 09:03:17,170 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 09:03:17,170 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.00 seconds 2025-02-14 09:03:17,170 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:03:17,170 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18518.85 MB 2025-02-14 09:03:17,170 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31862.85 MB 2025-02-14 09:03:17,170 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13344.00 MB 2025-02-14 09:03:17,170 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55394.17 MB 2025-02-14 09:03:17,170 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36985.37 MB 2025-02-14 09:03:17,170 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18408.80 MB 2025-02-14 09:03:17,170 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31898.72 MB 2025-02-14 09:03:17,458 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 09:03:17,458 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 09:03:17,458 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.28 seconds 2025-02-14 09:03:17,458 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:03:17,458 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31862.85 MB 2025-02-14 09:03:17,458 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23504.57 MB 2025-02-14 09:03:17,458 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8358.28 MB 2025-02-14 09:03:17,458 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36985.37 MB 2025-02-14 09:03:17,458 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36985.37 MB 2025-02-14 09:03:17,458 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:03:17,458 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34359.46 MB 2025-02-14 09:03:17,477 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8113, cut from 8115 2025-02-14 09:03:17,477 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 09:03:17,484 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 09:03:17,484 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 09:03:17,484 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:03:17,484 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:03:17,484 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23504.57 MB 2025-02-14 09:03:17,484 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31892.68 MB 2025-02-14 09:03:17,484 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8388.11 MB 2025-02-14 09:03:17,484 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36985.37 MB 2025-02-14 09:03:17,484 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41156.61 MB 2025-02-14 09:03:17,484 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4171.24 MB 2025-02-14 09:03:17,484 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31892.68 MB 2025-02-14 09:03:17,722 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7905] 2025-02-14 09:03:17,723 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:03:17,723 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 09:03:17,724 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:03:17,724 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 09:03:17,730 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 09:03:17,731 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:03:17,731 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 09:03:17,731 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 09:04:20,746 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:04:20,746 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 09:04:20,751 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 09:04:20,756 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:04:20,756 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1007, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 09:04:20,757 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:04:20,758 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1007, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 09:04:36,308 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 09:04:36,309 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 09:04:36,309 - resource_logging.py:150 - __exit__ - DEBUG - Time: 15.54 seconds 2025-02-14 09:04:36,309 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:04:36,309 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19985.64 MB 2025-02-14 09:04:36,309 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23549.36 MB 2025-02-14 09:04:36,309 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3563.72 MB 2025-02-14 09:04:36,309 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49494.88 MB 2025-02-14 09:04:36,309 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28812.77 MB 2025-02-14 09:04:36,309 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20682.11 MB 2025-02-14 09:04:36,309 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32402.22 MB 2025-02-14 09:04:36,372 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 09:04:36,372 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 09:04:36,372 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 09:04:36,372 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:04:36,372 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23549.36 MB 2025-02-14 09:04:36,372 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21012.93 MB 2025-02-14 09:04:36,372 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2536.43 MB 2025-02-14 09:04:36,372 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28812.77 MB 2025-02-14 09:04:36,372 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39130.76 MB 2025-02-14 09:04:36,372 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10317.99 MB 2025-02-14 09:04:36,372 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34106.76 MB 2025-02-14 09:04:38,275 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 09:04:38,275 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 09:04:38,275 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.90 seconds 2025-02-14 09:04:38,275 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:04:38,275 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21012.93 MB 2025-02-14 09:04:38,275 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21543.77 MB 2025-02-14 09:04:38,275 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 09:04:38,275 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39130.76 MB 2025-02-14 09:04:38,275 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26663.19 MB 2025-02-14 09:04:38,275 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12467.57 MB 2025-02-14 09:04:38,275 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25523.10 MB 2025-02-14 09:04:38,290 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 09:04:38,290 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 09:04:38,290 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:04:38,290 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:04:38,290 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21543.77 MB 2025-02-14 09:04:38,290 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23433.31 MB 2025-02-14 09:04:38,290 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 09:04:38,290 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26663.19 MB 2025-02-14 09:04:38,290 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28550.63 MB 2025-02-14 09:04:38,290 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 09:04:38,290 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24850.73 MB 2025-02-14 09:04:38,502 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 09:04:38,502 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 09:04:38,502 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 09:04:38,502 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:04:38,502 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23433.31 MB 2025-02-14 09:04:38,502 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25675.16 MB 2025-02-14 09:04:38,502 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 09:04:38,502 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28550.63 MB 2025-02-14 09:04:38,502 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34212.94 MB 2025-02-14 09:04:38,502 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 09:04:38,502 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31219.44 MB 2025-02-14 09:04:38,503 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 09:04:38,503 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 09:04:38,503 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 09:04:38,503 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:04:38,503 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21543.77 MB 2025-02-14 09:04:38,503 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25675.16 MB 2025-02-14 09:04:38,503 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 09:04:38,503 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26663.19 MB 2025-02-14 09:04:38,503 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34212.94 MB 2025-02-14 09:04:38,503 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-14 09:04:38,503 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31219.44 MB 2025-02-14 09:04:38,676 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 09:04:38,676 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 09:04:38,676 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 09:04:38,676 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:04:38,676 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27208.70 MB 2025-02-14 09:04:38,676 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27975.71 MB 2025-02-14 09:04:38,676 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 09:04:38,676 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34212.94 MB 2025-02-14 09:04:38,676 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34628.17 MB 2025-02-14 09:04:38,676 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 09:04:38,676 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28683.49 MB 2025-02-14 09:04:38,695 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 09:04:38,695 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 09:04:38,695 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:04:38,695 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:04:38,695 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28388.59 MB 2025-02-14 09:04:38,695 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28615.07 MB 2025-02-14 09:04:38,695 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.48 MB 2025-02-14 09:04:38,695 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34628.17 MB 2025-02-14 09:04:38,695 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34628.17 MB 2025-02-14 09:04:38,695 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:04:38,695 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28826.79 MB 2025-02-14 09:04:38,696 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 09:04:38,696 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 09:04:38,696 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.94 seconds 2025-02-14 09:04:38,696 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:04:38,696 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16477.17 MB 2025-02-14 09:04:38,696 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28815.43 MB 2025-02-14 09:04:38,696 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12338.26 MB 2025-02-14 09:04:38,696 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49494.88 MB 2025-02-14 09:04:38,696 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34628.17 MB 2025-02-14 09:04:38,696 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14866.71 MB 2025-02-14 09:04:38,696 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28826.79 MB 2025-02-14 09:04:38,963 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 09:04:38,963 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 09:04:38,963 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 09:04:38,963 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:04:38,963 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28815.43 MB 2025-02-14 09:04:38,963 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21470.52 MB 2025-02-14 09:04:38,963 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7344.92 MB 2025-02-14 09:04:38,963 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34628.17 MB 2025-02-14 09:04:38,963 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34628.17 MB 2025-02-14 09:04:38,963 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:04:38,963 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31318.19 MB 2025-02-14 09:04:38,981 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8133, cut from 8135 2025-02-14 09:04:38,982 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 09:04:38,988 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 09:04:38,988 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 09:04:38,988 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:04:38,988 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:04:38,988 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21470.52 MB 2025-02-14 09:04:38,988 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29879.82 MB 2025-02-14 09:04:38,988 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8409.30 MB 2025-02-14 09:04:38,988 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34628.17 MB 2025-02-14 09:04:38,988 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45078.28 MB 2025-02-14 09:04:38,988 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10450.11 MB 2025-02-14 09:04:38,988 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29879.82 MB 2025-02-14 09:04:39,157 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7925] 2025-02-14 09:04:39,158 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:04:39,158 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 09:04:39,159 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:04:39,159 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 09:04:39,164 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 09:04:39,165 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:04:39,165 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 09:04:39,165 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 09:05:09,984 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:05:09,984 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 09:05:09,989 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 09:05:09,992 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:05:09,992 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1285, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 09:05:09,993 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:05:09,993 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1285, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 09:05:29,734 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 09:05:29,734 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 09:05:29,734 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.73 seconds 2025-02-14 09:05:29,734 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:05:29,734 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21922.79 MB 2025-02-14 09:05:29,734 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26470.34 MB 2025-02-14 09:05:29,734 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4547.54 MB 2025-02-14 09:05:29,734 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53437.53 MB 2025-02-14 09:05:29,734 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38155.58 MB 2025-02-14 09:05:29,734 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15281.95 MB 2025-02-14 09:05:29,734 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35471.03 MB 2025-02-14 09:05:29,805 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 09:05:29,805 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 09:05:29,805 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 09:05:29,805 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:05:29,805 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26470.34 MB 2025-02-14 09:05:29,805 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22458.17 MB 2025-02-14 09:05:29,805 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4012.17 MB 2025-02-14 09:05:29,805 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38155.58 MB 2025-02-14 09:05:29,805 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46854.57 MB 2025-02-14 09:05:29,805 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8698.99 MB 2025-02-14 09:05:29,805 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39564.81 MB 2025-02-14 09:05:31,709 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 09:05:31,709 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 09:05:31,709 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.90 seconds 2025-02-14 09:05:31,709 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:05:31,709 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22458.17 MB 2025-02-14 09:05:31,709 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22989.01 MB 2025-02-14 09:05:31,709 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 09:05:31,709 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46854.57 MB 2025-02-14 09:05:31,709 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33606.86 MB 2025-02-14 09:05:31,709 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13247.71 MB 2025-02-14 09:05:31,709 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26968.34 MB 2025-02-14 09:05:31,722 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 09:05:31,722 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 09:05:31,722 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:05:31,722 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:05:31,722 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22989.01 MB 2025-02-14 09:05:31,722 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24878.54 MB 2025-02-14 09:05:31,722 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 09:05:31,722 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33606.86 MB 2025-02-14 09:05:31,722 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33606.86 MB 2025-02-14 09:05:31,722 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:05:31,722 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26295.97 MB 2025-02-14 09:05:31,933 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 09:05:31,933 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 09:05:31,933 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 09:05:31,933 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:05:31,933 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24878.54 MB 2025-02-14 09:05:31,933 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27120.40 MB 2025-02-14 09:05:31,933 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 09:05:31,933 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33606.86 MB 2025-02-14 09:05:31,933 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35966.16 MB 2025-02-14 09:05:31,933 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2359.30 MB 2025-02-14 09:05:31,933 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32664.68 MB 2025-02-14 09:05:31,933 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 09:05:31,933 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 09:05:31,934 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 09:05:31,934 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:05:31,934 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22989.01 MB 2025-02-14 09:05:31,934 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27120.40 MB 2025-02-14 09:05:31,934 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 09:05:31,934 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33606.86 MB 2025-02-14 09:05:31,934 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35966.16 MB 2025-02-14 09:05:31,934 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2359.30 MB 2025-02-14 09:05:31,934 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32664.68 MB 2025-02-14 09:05:32,101 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 09:05:32,101 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 09:05:32,101 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 09:05:32,101 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:05:32,102 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28653.94 MB 2025-02-14 09:05:32,102 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29420.94 MB 2025-02-14 09:05:32,102 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 09:05:32,102 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35966.16 MB 2025-02-14 09:05:32,102 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36381.39 MB 2025-02-14 09:05:32,102 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 09:05:32,102 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30128.73 MB 2025-02-14 09:05:32,120 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 09:05:32,120 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 09:05:32,120 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:05:32,120 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:05:32,120 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29833.83 MB 2025-02-14 09:05:32,120 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30062.72 MB 2025-02-14 09:05:32,120 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.89 MB 2025-02-14 09:05:32,120 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36381.39 MB 2025-02-14 09:05:32,120 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36381.39 MB 2025-02-14 09:05:32,120 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:05:32,121 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30284.72 MB 2025-02-14 09:05:32,122 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 09:05:32,122 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 09:05:32,122 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.13 seconds 2025-02-14 09:05:32,122 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:05:32,122 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17445.75 MB 2025-02-14 09:05:32,122 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30263.52 MB 2025-02-14 09:05:32,122 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12817.77 MB 2025-02-14 09:05:32,122 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53437.53 MB 2025-02-14 09:05:32,122 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36381.39 MB 2025-02-14 09:05:32,122 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17056.14 MB 2025-02-14 09:05:32,122 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30284.72 MB 2025-02-14 09:05:32,388 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 09:05:32,388 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 09:05:32,388 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 09:05:32,388 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:05:32,388 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30263.52 MB 2025-02-14 09:05:32,388 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22445.95 MB 2025-02-14 09:05:32,388 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7817.57 MB 2025-02-14 09:05:32,388 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36381.39 MB 2025-02-14 09:05:32,388 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36381.39 MB 2025-02-14 09:05:32,388 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:05:32,388 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32771.81 MB 2025-02-14 09:05:32,406 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8151, cut from 8153 2025-02-14 09:05:32,406 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 09:05:32,412 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 09:05:32,412 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 09:05:32,412 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:05:32,412 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:05:32,412 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22445.95 MB 2025-02-14 09:05:32,412 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30873.28 MB 2025-02-14 09:05:32,412 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8427.34 MB 2025-02-14 09:05:32,412 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36381.39 MB 2025-02-14 09:05:32,412 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44761.61 MB 2025-02-14 09:05:32,412 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8380.22 MB 2025-02-14 09:05:32,412 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30873.28 MB 2025-02-14 09:05:32,575 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7943] 2025-02-14 09:05:32,576 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:05:32,576 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 09:05:32,577 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:05:32,577 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 09:05:32,582 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 09:05:32,583 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:05:32,583 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 09:05:32,583 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 09:07:05,207 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:07:05,207 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 09:07:05,212 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 09:07:05,216 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:07:05,216 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 811, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 09:07:05,217 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:07:05,217 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 811, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 09:07:17,579 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 09:07:17,579 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 09:07:17,579 - resource_logging.py:150 - __exit__ - DEBUG - Time: 12.36 seconds 2025-02-14 09:07:17,579 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:07:17,579 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18619.88 MB 2025-02-14 09:07:17,579 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21490.89 MB 2025-02-14 09:07:17,579 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2871.00 MB 2025-02-14 09:07:17,579 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53141.83 MB 2025-02-14 09:07:17,579 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28129.10 MB 2025-02-14 09:07:17,579 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25012.73 MB 2025-02-14 09:07:17,579 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30356.99 MB 2025-02-14 09:07:17,626 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 09:07:17,626 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 09:07:17,626 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-14 09:07:17,626 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:07:17,626 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21490.89 MB 2025-02-14 09:07:17,626 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19993.99 MB 2025-02-14 09:07:17,626 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1496.90 MB 2025-02-14 09:07:17,626 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28129.10 MB 2025-02-14 09:07:17,626 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34139.54 MB 2025-02-14 09:07:17,626 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6010.44 MB 2025-02-14 09:07:17,626 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30004.41 MB 2025-02-14 09:07:19,533 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 09:07:19,533 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 09:07:19,533 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 09:07:19,533 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:07:19,533 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19993.99 MB 2025-02-14 09:07:19,533 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20524.83 MB 2025-02-14 09:07:19,533 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 09:07:19,533 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34139.54 MB 2025-02-14 09:07:19,533 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26673.68 MB 2025-02-14 09:07:19,533 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7465.86 MB 2025-02-14 09:07:19,533 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24504.16 MB 2025-02-14 09:07:19,546 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 09:07:19,546 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 09:07:19,546 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:07:19,546 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:07:19,546 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20524.83 MB 2025-02-14 09:07:19,546 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22414.36 MB 2025-02-14 09:07:19,546 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 09:07:19,546 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26673.68 MB 2025-02-14 09:07:19,546 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26673.68 MB 2025-02-14 09:07:19,546 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:07:19,546 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23831.79 MB 2025-02-14 09:07:19,756 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 09:07:19,756 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 09:07:19,756 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 09:07:19,756 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:07:19,756 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22414.36 MB 2025-02-14 09:07:19,756 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24656.22 MB 2025-02-14 09:07:19,756 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 09:07:19,756 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26673.68 MB 2025-02-14 09:07:19,756 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32807.85 MB 2025-02-14 09:07:19,756 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 09:07:19,756 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30200.50 MB 2025-02-14 09:07:19,757 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 09:07:19,757 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 09:07:19,757 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 09:07:19,757 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:07:19,757 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20524.83 MB 2025-02-14 09:07:19,757 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24656.22 MB 2025-02-14 09:07:19,757 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 09:07:19,757 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26673.68 MB 2025-02-14 09:07:19,757 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32807.85 MB 2025-02-14 09:07:19,757 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 09:07:19,757 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30200.50 MB 2025-02-14 09:07:19,924 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 09:07:19,925 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 09:07:19,925 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 09:07:19,925 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:07:19,925 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26189.76 MB 2025-02-14 09:07:19,925 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26956.76 MB 2025-02-14 09:07:19,925 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 09:07:19,925 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32807.85 MB 2025-02-14 09:07:19,925 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33223.08 MB 2025-02-14 09:07:19,925 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 09:07:19,925 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27664.55 MB 2025-02-14 09:07:19,943 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 09:07:19,944 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 09:07:19,944 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:07:19,944 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:07:19,944 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27369.65 MB 2025-02-14 09:07:19,944 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27601.03 MB 2025-02-14 09:07:19,944 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 231.38 MB 2025-02-14 09:07:19,944 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33223.08 MB 2025-02-14 09:07:19,944 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33223.08 MB 2025-02-14 09:07:19,944 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:07:19,944 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27798.44 MB 2025-02-14 09:07:19,945 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 09:07:19,945 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 09:07:19,945 - resource_logging.py:150 - __exit__ - DEBUG - Time: 14.73 seconds 2025-02-14 09:07:19,945 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:07:19,945 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15794.30 MB 2025-02-14 09:07:19,945 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27802.11 MB 2025-02-14 09:07:19,945 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12007.81 MB 2025-02-14 09:07:19,945 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53141.83 MB 2025-02-14 09:07:19,945 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33223.08 MB 2025-02-14 09:07:19,945 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19918.75 MB 2025-02-14 09:07:19,945 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27802.11 MB 2025-02-14 09:07:20,212 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 09:07:20,212 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 09:07:20,212 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 09:07:20,212 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:07:20,212 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27802.11 MB 2025-02-14 09:07:20,212 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20798.68 MB 2025-02-14 09:07:20,212 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7003.42 MB 2025-02-14 09:07:20,212 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33223.08 MB 2025-02-14 09:07:20,212 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33223.08 MB 2025-02-14 09:07:20,212 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:07:20,212 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30313.77 MB 2025-02-14 09:07:20,230 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 09:07:20,230 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 09:07:20,236 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 09:07:20,236 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 09:07:20,236 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:07:20,236 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:07:20,236 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20798.68 MB 2025-02-14 09:07:20,236 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29237.71 MB 2025-02-14 09:07:20,236 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 09:07:20,236 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33223.08 MB 2025-02-14 09:07:20,236 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41613.79 MB 2025-02-14 09:07:20,236 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 09:07:20,236 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29237.71 MB 2025-02-14 09:07:20,400 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 09:07:20,401 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:07:20,401 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 09:07:20,402 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:07:20,402 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 09:07:20,407 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 09:07:20,408 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:07:20,408 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 09:07:20,408 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 09:10:00,930 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:10:00,930 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 09:10:00,939 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 09:10:00,946 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:10:00,946 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1801, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 09:10:00,948 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:10:00,948 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1801, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 09:10:28,582 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 09:10:28,582 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 09:10:28,582 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.62 seconds 2025-02-14 09:10:28,582 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:10:28,582 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25518.36 MB 2025-02-14 09:10:28,582 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31892.00 MB 2025-02-14 09:10:28,582 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6373.64 MB 2025-02-14 09:10:28,582 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54198.80 MB 2025-02-14 09:10:28,582 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40024.15 MB 2025-02-14 09:10:28,582 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14174.65 MB 2025-02-14 09:10:28,582 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40879.34 MB 2025-02-14 09:10:28,689 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 09:10:28,689 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 09:10:28,689 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 09:10:28,689 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:10:28,689 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31892.00 MB 2025-02-14 09:10:28,689 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25140.69 MB 2025-02-14 09:10:28,689 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6751.31 MB 2025-02-14 09:10:28,689 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40024.15 MB 2025-02-14 09:10:28,689 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 59844.33 MB 2025-02-14 09:10:28,689 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 19820.18 MB 2025-02-14 09:10:28,689 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50566.44 MB 2025-02-14 09:10:30,604 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 09:10:30,604 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 09:10:30,604 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 09:10:30,604 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:10:30,604 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25140.69 MB 2025-02-14 09:10:30,604 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25671.53 MB 2025-02-14 09:10:30,604 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 09:10:30,604 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59844.33 MB 2025-02-14 09:10:30,604 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35064.38 MB 2025-02-14 09:10:30,604 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24779.95 MB 2025-02-14 09:10:30,604 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29650.86 MB 2025-02-14 09:10:30,618 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 09:10:30,618 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 09:10:30,618 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:10:30,618 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:10:30,618 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25671.53 MB 2025-02-14 09:10:30,618 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27561.06 MB 2025-02-14 09:10:30,618 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 09:10:30,618 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35064.38 MB 2025-02-14 09:10:30,618 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35064.38 MB 2025-02-14 09:10:30,618 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:10:30,618 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28978.49 MB 2025-02-14 09:10:30,834 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 09:10:30,834 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 09:10:30,834 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 09:10:30,834 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:10:30,834 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27561.06 MB 2025-02-14 09:10:30,834 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29802.92 MB 2025-02-14 09:10:30,834 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 09:10:30,834 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35064.38 MB 2025-02-14 09:10:30,834 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38367.40 MB 2025-02-14 09:10:30,834 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 09:10:30,834 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35347.20 MB 2025-02-14 09:10:30,835 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 09:10:30,835 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 09:10:30,835 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 09:10:30,835 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:10:30,835 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25671.53 MB 2025-02-14 09:10:30,835 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29802.92 MB 2025-02-14 09:10:30,835 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 09:10:30,835 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35064.38 MB 2025-02-14 09:10:30,835 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38367.40 MB 2025-02-14 09:10:30,835 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 09:10:30,835 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35347.20 MB 2025-02-14 09:10:31,003 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 09:10:31,003 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 09:10:31,003 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 09:10:31,003 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:10:31,003 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31336.46 MB 2025-02-14 09:10:31,003 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32103.46 MB 2025-02-14 09:10:31,003 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 09:10:31,003 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38367.40 MB 2025-02-14 09:10:31,003 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38780.53 MB 2025-02-14 09:10:31,003 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 09:10:31,003 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32811.25 MB 2025-02-14 09:10:31,022 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 09:10:31,022 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 09:10:31,022 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:10:31,022 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:10:31,022 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32516.35 MB 2025-02-14 09:10:31,022 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32743.72 MB 2025-02-14 09:10:31,022 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.36 MB 2025-02-14 09:10:31,022 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38780.53 MB 2025-02-14 09:10:31,022 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38780.53 MB 2025-02-14 09:10:31,022 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:10:31,022 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32985.82 MB 2025-02-14 09:10:31,023 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 09:10:31,023 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 09:10:31,023 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.07 seconds 2025-02-14 09:10:31,023 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:10:31,023 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19243.53 MB 2025-02-14 09:10:31,023 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32943.73 MB 2025-02-14 09:10:31,023 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13700.20 MB 2025-02-14 09:10:31,023 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54198.80 MB 2025-02-14 09:10:31,023 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38780.53 MB 2025-02-14 09:10:31,023 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15418.26 MB 2025-02-14 09:10:31,023 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32985.82 MB 2025-02-14 09:10:31,289 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 09:10:31,289 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 09:10:31,289 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 09:10:31,289 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:10:31,289 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32943.73 MB 2025-02-14 09:10:31,289 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24231.54 MB 2025-02-14 09:10:31,289 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8712.19 MB 2025-02-14 09:10:31,289 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38780.53 MB 2025-02-14 09:10:31,289 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38780.53 MB 2025-02-14 09:10:31,289 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:10:31,289 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35442.19 MB 2025-02-14 09:10:31,306 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8119, cut from 8121 2025-02-14 09:10:31,307 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 09:10:31,313 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 09:10:31,313 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 09:10:31,313 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:10:31,313 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:10:31,313 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24231.54 MB 2025-02-14 09:10:31,313 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32625.85 MB 2025-02-14 09:10:31,313 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8394.31 MB 2025-02-14 09:10:31,313 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38780.53 MB 2025-02-14 09:10:31,313 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42953.87 MB 2025-02-14 09:10:31,313 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4173.33 MB 2025-02-14 09:10:31,313 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32625.85 MB 2025-02-14 09:10:31,475 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7911] 2025-02-14 09:10:31,477 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:10:31,477 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 09:10:31,478 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:10:31,478 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 09:10:31,484 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 09:10:31,485 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:10:31,485 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 09:10:31,485 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 09:10:38,142 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:10:38,143 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 09:10:38,147 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 09:10:38,151 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:10:38,151 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 3309, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 09:10:38,152 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:10:38,152 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 3309, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 09:11:29,744 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 09:11:29,744 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 09:11:29,744 - resource_logging.py:150 - __exit__ - DEBUG - Time: 51.58 seconds 2025-02-14 09:11:29,744 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:11:29,744 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36026.48 MB 2025-02-14 09:11:29,744 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47736.98 MB 2025-02-14 09:11:29,744 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11710.50 MB 2025-02-14 09:11:29,744 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 74360.82 MB 2025-02-14 09:11:29,744 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51680.12 MB 2025-02-14 09:11:29,744 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22680.70 MB 2025-02-14 09:11:29,744 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 59447.34 MB 2025-02-14 09:11:29,964 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 09:11:29,965 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 09:11:29,965 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 09:11:29,965 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:11:29,965 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47736.98 MB 2025-02-14 09:11:29,965 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32980.37 MB 2025-02-14 09:11:29,965 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -14756.60 MB 2025-02-14 09:11:29,965 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51680.12 MB 2025-02-14 09:11:29,965 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 96007.62 MB 2025-02-14 09:11:29,965 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 44327.50 MB 2025-02-14 09:11:29,965 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 81396.60 MB 2025-02-14 09:11:31,948 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 09:11:31,948 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 09:11:31,948 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.98 seconds 2025-02-14 09:11:31,948 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:11:31,948 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32980.37 MB 2025-02-14 09:11:31,948 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33511.22 MB 2025-02-14 09:11:31,948 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 09:11:31,948 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 96007.62 MB 2025-02-14 09:11:31,948 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35529.95 MB 2025-02-14 09:11:31,948 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -60477.67 MB 2025-02-14 09:11:31,948 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37491.59 MB 2025-02-14 09:11:31,962 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 09:11:31,962 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 09:11:31,962 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:11:31,962 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:11:31,962 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33511.22 MB 2025-02-14 09:11:31,962 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35400.75 MB 2025-02-14 09:11:31,962 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 09:11:31,962 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35529.95 MB 2025-02-14 09:11:31,962 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38832.96 MB 2025-02-14 09:11:31,962 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 09:11:31,962 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36818.18 MB 2025-02-14 09:11:32,172 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 09:11:32,172 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 09:11:32,172 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 09:11:32,172 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:11:32,172 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35400.75 MB 2025-02-14 09:11:32,172 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37642.61 MB 2025-02-14 09:11:32,172 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 09:11:32,172 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38832.96 MB 2025-02-14 09:11:32,172 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45438.99 MB 2025-02-14 09:11:32,172 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 09:11:32,172 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43186.89 MB 2025-02-14 09:11:32,173 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 09:11:32,173 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 09:11:32,173 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 09:11:32,173 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:11:32,173 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33511.22 MB 2025-02-14 09:11:32,173 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37642.61 MB 2025-02-14 09:11:32,173 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 09:11:32,173 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35529.95 MB 2025-02-14 09:11:32,173 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45438.99 MB 2025-02-14 09:11:32,173 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9909.04 MB 2025-02-14 09:11:32,173 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43186.89 MB 2025-02-14 09:11:32,339 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 09:11:32,339 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 09:11:32,339 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 09:11:32,339 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:11:32,339 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39176.15 MB 2025-02-14 09:11:32,339 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39943.15 MB 2025-02-14 09:11:32,339 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 09:11:32,339 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45438.99 MB 2025-02-14 09:11:32,339 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45854.23 MB 2025-02-14 09:11:32,339 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 09:11:32,339 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40650.94 MB 2025-02-14 09:11:32,358 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 09:11:32,358 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 09:11:32,358 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:11:32,358 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:11:32,358 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40356.04 MB 2025-02-14 09:11:32,358 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40584.63 MB 2025-02-14 09:11:32,358 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.59 MB 2025-02-14 09:11:32,358 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45854.23 MB 2025-02-14 09:11:32,358 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45854.23 MB 2025-02-14 09:11:32,358 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:11:32,358 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40805.23 MB 2025-02-14 09:11:32,359 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 09:11:32,359 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 09:11:32,359 - resource_logging.py:150 - __exit__ - DEBUG - Time: 54.21 seconds 2025-02-14 09:11:32,359 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:11:32,359 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24497.59 MB 2025-02-14 09:11:32,359 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40785.14 MB 2025-02-14 09:11:32,359 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 16287.55 MB 2025-02-14 09:11:32,359 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62830.67 MB 2025-02-14 09:11:32,359 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45854.23 MB 2025-02-14 09:11:32,360 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16976.45 MB 2025-02-14 09:11:32,360 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40805.23 MB 2025-02-14 09:11:32,630 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 09:11:32,630 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 09:11:32,630 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 09:11:32,630 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:11:32,630 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40785.14 MB 2025-02-14 09:11:32,630 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29493.22 MB 2025-02-14 09:11:32,630 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -11291.92 MB 2025-02-14 09:11:32,630 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45854.23 MB 2025-02-14 09:11:32,630 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45854.23 MB 2025-02-14 09:11:32,630 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:11:32,630 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43289.74 MB 2025-02-14 09:11:32,649 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8139, cut from 8141 2025-02-14 09:11:32,649 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 09:11:32,655 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 09:11:32,655 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 09:11:32,655 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:11:32,655 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:11:32,655 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29493.22 MB 2025-02-14 09:11:32,655 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37908.17 MB 2025-02-14 09:11:32,655 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8414.95 MB 2025-02-14 09:11:32,655 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45854.23 MB 2025-02-14 09:11:32,655 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50038.05 MB 2025-02-14 09:11:32,655 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4183.82 MB 2025-02-14 09:11:32,655 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37908.17 MB 2025-02-14 09:11:32,823 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7931] 2025-02-14 09:11:32,824 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:11:32,824 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 09:11:32,825 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:11:32,825 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 09:11:32,830 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 09:11:32,831 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:11:32,831 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 09:11:32,831 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 09:11:39,874 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:11:39,874 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 09:11:39,882 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 09:11:39,889 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:11:39,889 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 88, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 09:11:39,891 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:11:39,891 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 88, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 09:11:41,360 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 09:11:41,361 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 09:11:41,361 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.46 seconds 2025-02-14 09:11:41,361 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:11:41,361 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13581.90 MB 2025-02-14 09:11:41,361 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 13893.33 MB 2025-02-14 09:11:41,361 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 311.43 MB 2025-02-14 09:11:41,361 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58405.68 MB 2025-02-14 09:11:41,361 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 16911.43 MB 2025-02-14 09:11:41,361 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -41494.25 MB 2025-02-14 09:11:41,361 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22826.78 MB 2025-02-14 09:11:41,364 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 09:11:41,364 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 09:11:41,364 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 09:11:41,364 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:11:41,364 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13893.33 MB 2025-02-14 09:11:41,364 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14044.22 MB 2025-02-14 09:11:41,364 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 150.89 MB 2025-02-14 09:11:41,364 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 16911.43 MB 2025-02-14 09:11:41,364 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 16911.43 MB 2025-02-14 09:11:41,364 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:11:41,364 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14511.42 MB 2025-02-14 09:11:41,805 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 09:11:41,805 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 09:11:41,805 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.44 seconds 2025-02-14 09:11:41,805 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:11:41,805 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14044.22 MB 2025-02-14 09:11:41,805 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14161.00 MB 2025-02-14 09:11:41,805 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 116.79 MB 2025-02-14 09:11:41,805 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 16911.43 MB 2025-02-14 09:11:41,805 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 16911.43 MB 2025-02-14 09:11:41,805 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:11:41,805 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18130.76 MB 2025-02-14 09:11:41,810 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 09:11:41,811 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 09:11:41,811 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 09:11:41,811 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:11:41,811 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14160.94 MB 2025-02-14 09:11:41,811 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14576.53 MB 2025-02-14 09:11:41,811 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 415.60 MB 2025-02-14 09:11:41,811 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 16911.43 MB 2025-02-14 09:11:41,811 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 16911.43 MB 2025-02-14 09:11:41,811 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:11:41,811 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14888.37 MB 2025-02-14 09:11:41,899 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 09:11:41,899 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 09:11:41,899 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 09:11:41,900 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:11:41,900 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14576.53 MB 2025-02-14 09:11:41,900 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15081.33 MB 2025-02-14 09:11:41,900 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 504.79 MB 2025-02-14 09:11:41,900 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 16911.43 MB 2025-02-14 09:11:41,900 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 16911.43 MB 2025-02-14 09:11:41,900 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:11:41,900 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16289.48 MB 2025-02-14 09:11:41,900 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 09:11:41,900 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 09:11:41,900 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 09:11:41,900 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:11:41,900 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14160.94 MB 2025-02-14 09:11:41,900 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15081.33 MB 2025-02-14 09:11:41,900 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 920.39 MB 2025-02-14 09:11:41,900 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 16911.43 MB 2025-02-14 09:11:41,900 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 16911.43 MB 2025-02-14 09:11:41,900 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:11:41,900 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16289.48 MB 2025-02-14 09:11:41,946 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 09:11:41,946 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 09:11:41,946 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-14 09:11:41,946 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:11:41,946 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15568.65 MB 2025-02-14 09:11:41,946 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15780.65 MB 2025-02-14 09:11:41,946 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 211.99 MB 2025-02-14 09:11:41,946 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 16911.43 MB 2025-02-14 09:11:41,946 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17043.55 MB 2025-02-14 09:11:41,946 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 132.12 MB 2025-02-14 09:11:41,946 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15936.36 MB 2025-02-14 09:11:41,951 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 09:11:41,951 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 09:11:41,951 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 09:11:41,951 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:11:41,951 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15914.75 MB 2025-02-14 09:11:41,951 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16125.22 MB 2025-02-14 09:11:41,951 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 210.48 MB 2025-02-14 09:11:41,951 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17043.55 MB 2025-02-14 09:11:41,951 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17043.55 MB 2025-02-14 09:11:41,951 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:11:41,951 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16125.22 MB 2025-02-14 09:11:41,953 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 09:11:41,953 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 09:11:41,953 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.06 seconds 2025-02-14 09:11:41,953 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:11:41,953 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13275.31 MB 2025-02-14 09:11:41,953 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16312.84 MB 2025-02-14 09:11:41,953 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3037.54 MB 2025-02-14 09:11:41,953 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58405.68 MB 2025-02-14 09:11:41,953 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17043.55 MB 2025-02-14 09:11:41,953 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -41362.13 MB 2025-02-14 09:11:41,953 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16312.84 MB 2025-02-14 09:11:42,206 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 09:11:42,206 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 09:11:42,206 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.25 seconds 2025-02-14 09:11:42,206 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:11:42,206 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13786.50 MB 2025-02-14 09:11:42,206 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16598.88 MB 2025-02-14 09:11:42,206 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2812.39 MB 2025-02-14 09:11:42,206 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17043.55 MB 2025-02-14 09:11:42,206 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18176.02 MB 2025-02-14 09:11:42,206 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1132.46 MB 2025-02-14 09:11:42,206 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16880.93 MB 2025-02-14 09:11:42,224 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 7615, cut from 7617 2025-02-14 09:11:42,224 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 09:11:42,229 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 09:11:42,230 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 09:11:42,230 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:11:42,230 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:11:42,230 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16598.88 MB 2025-02-14 09:11:42,230 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24472.97 MB 2025-02-14 09:11:42,230 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7874.08 MB 2025-02-14 09:11:42,230 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18176.02 MB 2025-02-14 09:11:42,230 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27965.52 MB 2025-02-14 09:11:42,230 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9789.51 MB 2025-02-14 09:11:42,230 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24472.97 MB 2025-02-14 09:11:42,381 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7407] 2025-02-14 09:11:42,383 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:11:42,383 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 09:11:42,384 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:11:42,384 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 09:11:42,388 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 09:11:42,390 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:11:42,390 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 09:11:42,390 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 09:11:50,848 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:11:50,848 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 09:11:50,853 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 09:11:50,857 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:11:50,857 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 106, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 09:11:50,858 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:11:50,858 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 106, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 09:11:52,518 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 09:11:52,518 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 09:11:52,518 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.66 seconds 2025-02-14 09:11:52,518 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:11:52,518 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13707.33 MB 2025-02-14 09:11:52,518 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14082.46 MB 2025-02-14 09:11:52,518 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 375.13 MB 2025-02-14 09:11:52,518 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35796.29 MB 2025-02-14 09:11:52,518 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 16909.34 MB 2025-02-14 09:11:52,518 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18886.95 MB 2025-02-14 09:11:52,518 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22952.21 MB 2025-02-14 09:11:52,521 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 09:11:52,521 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 09:11:52,521 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 09:11:52,521 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:11:52,521 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14082.46 MB 2025-02-14 09:11:52,521 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14264.21 MB 2025-02-14 09:11:52,521 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 181.75 MB 2025-02-14 09:11:52,521 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 16909.34 MB 2025-02-14 09:11:52,521 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 16909.34 MB 2025-02-14 09:11:52,521 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:11:52,521 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14826.96 MB 2025-02-14 09:11:53,045 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 09:11:53,045 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 09:11:53,045 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.52 seconds 2025-02-14 09:11:53,045 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:11:53,045 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14264.21 MB 2025-02-14 09:11:53,045 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14404.82 MB 2025-02-14 09:11:53,045 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 140.61 MB 2025-02-14 09:11:53,045 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 16909.34 MB 2025-02-14 09:11:53,045 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 16909.34 MB 2025-02-14 09:11:53,045 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:11:53,045 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18350.75 MB 2025-02-14 09:11:53,051 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 09:11:53,051 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 09:11:53,051 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 09:11:53,051 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:11:53,051 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14404.82 MB 2025-02-14 09:11:53,051 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14905.42 MB 2025-02-14 09:11:53,051 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 500.60 MB 2025-02-14 09:11:53,051 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 16909.34 MB 2025-02-14 09:11:53,051 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 16909.34 MB 2025-02-14 09:11:53,051 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:11:53,051 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15281.05 MB 2025-02-14 09:11:53,159 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 09:11:53,159 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 09:11:53,159 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 09:11:53,159 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:11:53,159 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14905.42 MB 2025-02-14 09:11:53,159 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15513.45 MB 2025-02-14 09:11:53,159 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 608.03 MB 2025-02-14 09:11:53,159 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 16909.34 MB 2025-02-14 09:11:53,159 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17664.31 MB 2025-02-14 09:11:53,159 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 754.97 MB 2025-02-14 09:11:53,159 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16968.75 MB 2025-02-14 09:11:53,159 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 09:11:53,160 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 09:11:53,160 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 09:11:53,160 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:11:53,160 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14404.82 MB 2025-02-14 09:11:53,160 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15513.45 MB 2025-02-14 09:11:53,160 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1108.64 MB 2025-02-14 09:11:53,160 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 16909.34 MB 2025-02-14 09:11:53,160 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17664.31 MB 2025-02-14 09:11:53,160 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 754.97 MB 2025-02-14 09:11:53,160 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16968.75 MB 2025-02-14 09:11:53,214 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 09:11:53,214 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 09:11:53,214 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-14 09:11:53,214 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:11:53,214 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16101.38 MB 2025-02-14 09:11:53,214 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16356.73 MB 2025-02-14 09:11:53,214 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 255.36 MB 2025-02-14 09:11:53,214 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17664.31 MB 2025-02-14 09:11:53,214 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17825.79 MB 2025-02-14 09:11:53,214 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 161.48 MB 2025-02-14 09:11:53,214 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16544.30 MB 2025-02-14 09:11:53,220 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 09:11:53,220 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 09:11:53,220 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 09:11:53,220 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:11:53,220 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16518.26 MB 2025-02-14 09:11:53,220 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16745.80 MB 2025-02-14 09:11:53,220 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.54 MB 2025-02-14 09:11:53,220 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17825.79 MB 2025-02-14 09:11:53,220 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17825.79 MB 2025-02-14 09:11:53,220 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:11:53,220 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16745.80 MB 2025-02-14 09:11:53,221 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 09:11:53,221 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 09:11:53,221 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.36 seconds 2025-02-14 09:11:53,221 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:11:53,221 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13338.02 MB 2025-02-14 09:11:53,221 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16946.80 MB 2025-02-14 09:11:53,221 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3608.78 MB 2025-02-14 09:11:53,221 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35796.29 MB 2025-02-14 09:11:53,221 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17825.79 MB 2025-02-14 09:11:53,221 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17970.50 MB 2025-02-14 09:11:53,221 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16946.80 MB 2025-02-14 09:11:53,493 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 09:11:53,493 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 09:11:53,493 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 09:11:53,493 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:11:53,493 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16946.80 MB 2025-02-14 09:11:53,493 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19959.73 MB 2025-02-14 09:11:53,493 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3012.93 MB 2025-02-14 09:11:53,493 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17825.79 MB 2025-02-14 09:11:53,493 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21583.89 MB 2025-02-14 09:11:53,493 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3758.10 MB 2025-02-14 09:11:53,493 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20261.30 MB 2025-02-14 09:11:53,512 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8159, cut from 8161 2025-02-14 09:11:53,512 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 09:11:53,518 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 09:11:53,518 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 09:11:53,518 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:11:53,518 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:11:53,518 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17051.43 MB 2025-02-14 09:11:53,518 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25487.61 MB 2025-02-14 09:11:53,518 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8436.18 MB 2025-02-14 09:11:53,518 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21583.89 MB 2025-02-14 09:11:53,518 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32069.65 MB 2025-02-14 09:11:53,518 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10485.76 MB 2025-02-14 09:11:53,518 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25487.61 MB 2025-02-14 09:11:53,681 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7951] 2025-02-14 09:11:53,683 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:11:53,683 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 09:11:53,684 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:11:53,684 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 09:11:53,688 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 09:11:53,690 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:11:53,690 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 09:11:53,690 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 09:12:54,270 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:12:54,270 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 09:12:54,275 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 09:12:54,278 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:12:54,278 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 162, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 09:12:54,279 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:12:54,279 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 162, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 09:12:56,793 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 09:12:56,793 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 09:12:56,793 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.51 seconds 2025-02-14 09:12:56,793 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:12:56,793 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14097.55 MB 2025-02-14 09:12:56,793 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14670.86 MB 2025-02-14 09:12:56,793 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 573.31 MB 2025-02-14 09:12:56,793 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40458.26 MB 2025-02-14 09:12:56,793 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 16907.24 MB 2025-02-14 09:12:56,793 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23551.02 MB 2025-02-14 09:12:56,793 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23569.73 MB 2025-02-14 09:12:56,806 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 09:12:56,806 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 09:12:56,806 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:12:56,806 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:12:56,806 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14670.86 MB 2025-02-14 09:12:56,806 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14836.26 MB 2025-02-14 09:12:56,806 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 165.40 MB 2025-02-14 09:12:56,806 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 16907.24 MB 2025-02-14 09:12:56,806 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17943.23 MB 2025-02-14 09:12:56,806 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1035.99 MB 2025-02-14 09:12:56,806 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16728.72 MB 2025-02-14 09:12:57,545 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 09:12:57,545 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 09:12:57,545 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.74 seconds 2025-02-14 09:12:57,545 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:12:57,545 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14836.26 MB 2025-02-14 09:12:57,545 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15030.01 MB 2025-02-14 09:12:57,545 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 193.76 MB 2025-02-14 09:12:57,545 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17943.23 MB 2025-02-14 09:12:57,545 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17425.24 MB 2025-02-14 09:12:57,545 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -518.00 MB 2025-02-14 09:12:57,545 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19007.73 MB 2025-02-14 09:12:57,552 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 09:12:57,552 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 09:12:57,552 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 09:12:57,552 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:12:57,552 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15029.95 MB 2025-02-14 09:12:57,552 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15719.46 MB 2025-02-14 09:12:57,552 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 689.51 MB 2025-02-14 09:12:57,552 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17425.24 MB 2025-02-14 09:12:57,552 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17425.24 MB 2025-02-14 09:12:57,552 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:12:57,552 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16236.83 MB 2025-02-14 09:12:57,635 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 09:12:57,636 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 09:12:57,636 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 09:12:57,636 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:12:57,636 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15719.46 MB 2025-02-14 09:12:57,636 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16537.78 MB 2025-02-14 09:12:57,636 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 818.32 MB 2025-02-14 09:12:57,636 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17425.24 MB 2025-02-14 09:12:57,636 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19675.48 MB 2025-02-14 09:12:57,636 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2250.24 MB 2025-02-14 09:12:57,636 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18561.40 MB 2025-02-14 09:12:57,636 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 09:12:57,636 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 09:12:57,636 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 09:12:57,636 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:12:57,636 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15029.95 MB 2025-02-14 09:12:57,636 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16537.78 MB 2025-02-14 09:12:57,636 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1507.83 MB 2025-02-14 09:12:57,636 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17425.24 MB 2025-02-14 09:12:57,636 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19675.48 MB 2025-02-14 09:12:57,636 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2250.24 MB 2025-02-14 09:12:57,636 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18561.40 MB 2025-02-14 09:12:57,700 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 09:12:57,700 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 09:12:57,700 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 09:12:57,700 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:12:57,700 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17097.52 MB 2025-02-14 09:12:57,700 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17377.48 MB 2025-02-14 09:12:57,700 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 279.96 MB 2025-02-14 09:12:57,700 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19675.48 MB 2025-02-14 09:12:57,700 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19826.48 MB 2025-02-14 09:12:57,700 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 150.99 MB 2025-02-14 09:12:57,700 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17648.45 MB 2025-02-14 09:12:57,709 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 09:12:57,709 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 09:12:57,709 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:12:57,709 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:12:57,709 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17528.19 MB 2025-02-14 09:12:57,709 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17732.02 MB 2025-02-14 09:12:57,709 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 203.83 MB 2025-02-14 09:12:57,709 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19826.48 MB 2025-02-14 09:12:57,709 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19830.67 MB 2025-02-14 09:12:57,709 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-14 09:12:57,709 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17750.21 MB 2025-02-14 09:12:57,711 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 09:12:57,711 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 09:12:57,711 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.43 seconds 2025-02-14 09:12:57,711 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:12:57,711 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13533.13 MB 2025-02-14 09:12:57,711 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17932.82 MB 2025-02-14 09:12:57,711 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4399.69 MB 2025-02-14 09:12:57,711 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40458.26 MB 2025-02-14 09:12:57,711 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19830.67 MB 2025-02-14 09:12:57,711 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20627.59 MB 2025-02-14 09:12:57,711 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17932.82 MB 2025-02-14 09:12:57,977 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 09:12:57,977 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 09:12:57,977 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 09:12:57,978 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:12:57,978 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17932.82 MB 2025-02-14 09:12:57,978 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17334.10 MB 2025-02-14 09:12:57,978 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -598.72 MB 2025-02-14 09:12:57,978 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19830.67 MB 2025-02-14 09:12:57,978 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19830.67 MB 2025-02-14 09:12:57,978 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:12:57,978 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19036.47 MB 2025-02-14 09:12:57,995 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8151, cut from 8153 2025-02-14 09:12:57,996 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 09:12:58,002 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 09:12:58,002 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 09:12:58,002 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:12:58,002 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:12:58,002 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17334.10 MB 2025-02-14 09:12:58,002 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25761.44 MB 2025-02-14 09:12:58,002 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8427.34 MB 2025-02-14 09:12:58,002 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19830.67 MB 2025-02-14 09:12:58,002 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30305.94 MB 2025-02-14 09:12:58,002 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10475.27 MB 2025-02-14 09:12:58,002 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25761.44 MB 2025-02-14 09:12:58,170 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7943] 2025-02-14 09:12:58,171 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:12:58,171 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 09:12:58,173 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:12:58,173 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 09:12:58,178 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 09:12:58,179 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:12:58,179 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 09:12:58,179 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 09:13:12,297 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:13:12,298 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 09:13:12,303 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 09:13:12,306 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:13:12,306 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1077, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 09:13:12,307 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:13:12,307 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1077, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 09:13:28,920 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 09:13:28,921 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 09:13:28,921 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.61 seconds 2025-02-14 09:13:28,921 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:13:28,921 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20473.41 MB 2025-02-14 09:13:28,921 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24284.86 MB 2025-02-14 09:13:28,921 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3811.44 MB 2025-02-14 09:13:28,921 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38686.16 MB 2025-02-14 09:13:28,921 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31195.14 MB 2025-02-14 09:13:28,921 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7491.03 MB 2025-02-14 09:13:28,921 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33115.68 MB 2025-02-14 09:13:28,985 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 09:13:28,985 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 09:13:28,985 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 09:13:28,985 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:13:28,985 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24284.86 MB 2025-02-14 09:13:28,985 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21376.84 MB 2025-02-14 09:13:28,985 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2908.02 MB 2025-02-14 09:13:28,985 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31195.14 MB 2025-02-14 09:13:28,985 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40330.33 MB 2025-02-14 09:13:28,985 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9135.19 MB 2025-02-14 09:13:28,985 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35754.03 MB 2025-02-14 09:13:30,903 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 09:13:30,904 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 09:13:30,904 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 09:13:30,904 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:13:30,904 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21376.84 MB 2025-02-14 09:13:30,904 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21907.68 MB 2025-02-14 09:13:30,904 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 09:13:30,904 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40330.33 MB 2025-02-14 09:13:30,904 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28798.09 MB 2025-02-14 09:13:30,904 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11532.24 MB 2025-02-14 09:13:30,904 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25887.01 MB 2025-02-14 09:13:30,917 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 09:13:30,917 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 09:13:30,917 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:13:30,917 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:13:30,917 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21907.68 MB 2025-02-14 09:13:30,917 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23797.21 MB 2025-02-14 09:13:30,917 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 09:13:30,917 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28798.09 MB 2025-02-14 09:13:30,917 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28798.09 MB 2025-02-14 09:13:30,917 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:13:30,917 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25214.64 MB 2025-02-14 09:13:31,129 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 09:13:31,129 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 09:13:31,129 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 09:13:31,129 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:13:31,129 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23797.21 MB 2025-02-14 09:13:31,129 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26039.07 MB 2025-02-14 09:13:31,129 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 09:13:31,129 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28798.09 MB 2025-02-14 09:13:31,129 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34460.40 MB 2025-02-14 09:13:31,129 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 09:13:31,129 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31583.35 MB 2025-02-14 09:13:31,130 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 09:13:31,130 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 09:13:31,130 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 09:13:31,130 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:13:31,130 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21907.68 MB 2025-02-14 09:13:31,130 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26039.07 MB 2025-02-14 09:13:31,130 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 09:13:31,130 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28798.09 MB 2025-02-14 09:13:31,130 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34460.40 MB 2025-02-14 09:13:31,130 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 09:13:31,130 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31583.35 MB 2025-02-14 09:13:31,299 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 09:13:31,299 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 09:13:31,299 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 09:13:31,299 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:13:31,299 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27572.61 MB 2025-02-14 09:13:31,299 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28339.61 MB 2025-02-14 09:13:31,299 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 09:13:31,299 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34460.40 MB 2025-02-14 09:13:31,299 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34877.73 MB 2025-02-14 09:13:31,299 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 09:13:31,299 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29047.40 MB 2025-02-14 09:13:31,318 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 09:13:31,318 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 09:13:31,318 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:13:31,318 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:13:31,318 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28752.50 MB 2025-02-14 09:13:31,318 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28980.53 MB 2025-02-14 09:13:31,318 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.03 MB 2025-02-14 09:13:31,318 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34877.73 MB 2025-02-14 09:13:31,318 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34877.73 MB 2025-02-14 09:13:31,318 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:13:31,318 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29196.02 MB 2025-02-14 09:13:31,319 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 09:13:31,319 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 09:13:31,319 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.01 seconds 2025-02-14 09:13:31,319 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:13:31,319 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16721.06 MB 2025-02-14 09:13:31,319 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29181.06 MB 2025-02-14 09:13:31,319 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12460.00 MB 2025-02-14 09:13:31,319 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38686.16 MB 2025-02-14 09:13:31,319 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34877.73 MB 2025-02-14 09:13:31,319 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3808.43 MB 2025-02-14 09:13:31,319 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29196.02 MB 2025-02-14 09:13:31,588 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 09:13:31,588 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 09:13:31,588 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 09:13:31,588 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:13:31,588 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29181.06 MB 2025-02-14 09:13:31,588 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21717.07 MB 2025-02-14 09:13:31,588 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7463.99 MB 2025-02-14 09:13:31,588 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34877.73 MB 2025-02-14 09:13:31,588 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34877.73 MB 2025-02-14 09:13:31,588 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:13:31,588 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31685.97 MB 2025-02-14 09:13:31,606 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8140, cut from 8142 2025-02-14 09:13:31,606 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 09:13:31,612 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 09:13:31,612 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 09:13:31,612 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:13:31,612 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:13:31,612 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21717.07 MB 2025-02-14 09:13:31,612 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30133.67 MB 2025-02-14 09:13:31,612 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8416.60 MB 2025-02-14 09:13:31,612 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34877.73 MB 2025-02-14 09:13:31,612 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43245.37 MB 2025-02-14 09:13:31,612 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8367.64 MB 2025-02-14 09:13:31,612 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30133.67 MB 2025-02-14 09:13:31,773 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7932] 2025-02-14 09:13:31,775 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:13:31,775 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 09:13:31,776 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:13:31,776 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 09:13:31,780 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 09:13:31,781 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:13:31,782 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 09:13:31,782 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 09:15:06,530 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:15:06,530 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 09:15:06,535 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 09:15:06,538 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:15:06,538 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 248, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 09:15:06,539 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:15:06,539 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 248, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 09:15:10,344 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 09:15:10,344 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 09:15:10,344 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.80 seconds 2025-02-14 09:15:10,344 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:15:10,344 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14696.81 MB 2025-02-14 09:15:10,344 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15574.47 MB 2025-02-14 09:15:10,344 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 877.66 MB 2025-02-14 09:15:10,344 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51613.01 MB 2025-02-14 09:15:10,344 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17853.05 MB 2025-02-14 09:15:10,344 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33759.95 MB 2025-02-14 09:15:10,344 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24395.48 MB 2025-02-14 09:15:10,361 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 09:15:10,362 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 09:15:10,362 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:15:10,362 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:15:10,362 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15574.47 MB 2025-02-14 09:15:10,362 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15739.77 MB 2025-02-14 09:15:10,362 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 165.31 MB 2025-02-14 09:15:10,362 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17853.05 MB 2025-02-14 09:15:10,362 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19713.23 MB 2025-02-14 09:15:10,362 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1860.17 MB 2025-02-14 09:15:10,362 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18567.52 MB 2025-02-14 09:15:11,385 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 09:15:11,385 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 09:15:11,385 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.02 seconds 2025-02-14 09:15:11,385 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:15:11,385 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15739.77 MB 2025-02-14 09:15:11,385 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16019.79 MB 2025-02-14 09:15:11,385 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 280.02 MB 2025-02-14 09:15:11,385 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19713.23 MB 2025-02-14 09:15:11,385 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18532.53 MB 2025-02-14 09:15:11,385 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1180.70 MB 2025-02-14 09:15:11,385 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19996.18 MB 2025-02-14 09:15:11,394 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 09:15:11,394 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 09:15:11,394 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:15:11,394 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:15:11,394 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16019.79 MB 2025-02-14 09:15:11,394 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17016.28 MB 2025-02-14 09:15:11,394 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 996.49 MB 2025-02-14 09:15:11,394 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18532.53 MB 2025-02-14 09:15:11,394 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19031.65 MB 2025-02-14 09:15:11,394 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 499.12 MB 2025-02-14 09:15:11,394 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17763.98 MB 2025-02-14 09:15:11,508 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 09:15:11,508 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 09:15:11,508 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 09:15:11,508 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:15:11,508 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17016.28 MB 2025-02-14 09:15:11,508 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18199.55 MB 2025-02-14 09:15:11,508 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1183.27 MB 2025-02-14 09:15:11,508 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19031.65 MB 2025-02-14 09:15:11,508 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22525.51 MB 2025-02-14 09:15:11,508 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3493.86 MB 2025-02-14 09:15:11,508 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21124.78 MB 2025-02-14 09:15:11,509 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 09:15:11,509 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 09:15:11,509 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 09:15:11,509 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:15:11,509 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16019.79 MB 2025-02-14 09:15:11,509 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18199.55 MB 2025-02-14 09:15:11,509 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2179.75 MB 2025-02-14 09:15:11,509 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18532.53 MB 2025-02-14 09:15:11,509 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22525.51 MB 2025-02-14 09:15:11,509 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3992.98 MB 2025-02-14 09:15:11,509 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21124.78 MB 2025-02-14 09:15:11,598 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 09:15:11,598 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 09:15:11,598 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 09:15:11,598 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:15:11,598 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19008.49 MB 2025-02-14 09:15:11,598 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19413.74 MB 2025-02-14 09:15:11,598 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 405.25 MB 2025-02-14 09:15:11,598 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22525.51 MB 2025-02-14 09:15:11,598 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22743.61 MB 2025-02-14 09:15:11,598 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 218.10 MB 2025-02-14 09:15:11,598 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19789.81 MB 2025-02-14 09:15:11,610 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 09:15:11,610 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 09:15:11,610 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:15:11,610 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:15:11,610 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19631.54 MB 2025-02-14 09:15:11,610 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19839.81 MB 2025-02-14 09:15:11,610 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 208.27 MB 2025-02-14 09:15:11,610 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22743.61 MB 2025-02-14 09:15:11,610 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22743.61 MB 2025-02-14 09:15:11,610 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:15:11,610 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19878.33 MB 2025-02-14 09:15:11,611 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 09:15:11,611 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 09:15:11,611 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.07 seconds 2025-02-14 09:15:11,611 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:15:11,611 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13832.76 MB 2025-02-14 09:15:11,611 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20040.25 MB 2025-02-14 09:15:11,611 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6207.49 MB 2025-02-14 09:15:11,611 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51613.01 MB 2025-02-14 09:15:11,611 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22743.61 MB 2025-02-14 09:15:11,611 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28869.39 MB 2025-02-14 09:15:11,611 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20040.25 MB 2025-02-14 09:15:11,877 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 09:15:11,877 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 09:15:11,877 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 09:15:11,877 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:15:11,877 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14931.37 MB 2025-02-14 09:15:11,877 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17935.82 MB 2025-02-14 09:15:11,877 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3004.45 MB 2025-02-14 09:15:11,877 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22743.61 MB 2025-02-14 09:15:11,877 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22743.61 MB 2025-02-14 09:15:11,877 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:15:11,877 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18236.23 MB 2025-02-14 09:15:11,894 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8136, cut from 8138 2025-02-14 09:15:11,895 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 09:15:11,901 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 09:15:11,901 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 09:15:11,901 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:15:11,901 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:15:11,901 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17935.82 MB 2025-02-14 09:15:11,901 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26348.25 MB 2025-02-14 09:15:11,901 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8412.43 MB 2025-02-14 09:15:11,901 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22743.61 MB 2025-02-14 09:15:11,901 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33197.92 MB 2025-02-14 09:15:11,901 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10454.30 MB 2025-02-14 09:15:11,901 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26348.25 MB 2025-02-14 09:15:12,063 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7928] 2025-02-14 09:15:12,064 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:15:12,064 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 09:15:12,065 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:15:12,065 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 09:15:12,070 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 09:15:12,071 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:15:12,071 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 09:15:12,071 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 09:15:25,609 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:15:25,609 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 09:15:25,614 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 09:15:25,617 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:15:25,617 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2007, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 09:15:25,618 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:15:25,618 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2007, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 09:15:56,610 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 09:15:56,610 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 09:15:56,610 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.98 seconds 2025-02-14 09:15:56,610 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:15:56,610 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26953.80 MB 2025-02-14 09:15:56,610 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34056.86 MB 2025-02-14 09:15:56,610 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7103.05 MB 2025-02-14 09:15:56,610 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41561.36 MB 2025-02-14 09:15:56,610 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40735.08 MB 2025-02-14 09:15:56,610 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -826.28 MB 2025-02-14 09:15:56,610 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42995.20 MB 2025-02-14 09:15:56,735 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 09:15:56,735 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 09:15:56,735 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 09:15:56,735 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:15:56,735 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34056.86 MB 2025-02-14 09:15:56,735 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26211.62 MB 2025-02-14 09:15:56,735 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7845.24 MB 2025-02-14 09:15:56,735 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40735.08 MB 2025-02-14 09:15:56,735 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 64298.68 MB 2025-02-14 09:15:56,735 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 23563.60 MB 2025-02-14 09:15:56,735 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54417.31 MB 2025-02-14 09:15:58,677 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 09:15:58,677 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 09:15:58,677 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-14 09:15:58,678 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:15:58,678 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26211.62 MB 2025-02-14 09:15:58,678 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26742.46 MB 2025-02-14 09:15:58,678 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 09:15:58,678 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64298.68 MB 2025-02-14 09:15:58,678 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30865.88 MB 2025-02-14 09:15:58,678 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33432.80 MB 2025-02-14 09:15:58,678 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30722.83 MB 2025-02-14 09:15:58,691 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 09:15:58,691 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 09:15:58,691 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:15:58,691 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:15:58,691 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26742.46 MB 2025-02-14 09:15:58,691 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28631.99 MB 2025-02-14 09:15:58,691 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 09:15:58,691 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30865.88 MB 2025-02-14 09:15:58,691 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32753.32 MB 2025-02-14 09:15:58,691 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 09:15:58,691 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30049.42 MB 2025-02-14 09:15:58,899 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 09:15:58,899 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 09:15:58,899 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 09:15:58,899 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:15:58,899 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28631.99 MB 2025-02-14 09:15:58,899 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30873.85 MB 2025-02-14 09:15:58,900 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 09:15:58,900 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32753.32 MB 2025-02-14 09:15:58,900 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38889.59 MB 2025-02-14 09:15:58,900 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6136.27 MB 2025-02-14 09:15:58,900 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36418.13 MB 2025-02-14 09:15:58,900 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 09:15:58,900 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 09:15:58,900 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 09:15:58,900 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:15:58,900 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26742.46 MB 2025-02-14 09:15:58,900 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30873.85 MB 2025-02-14 09:15:58,900 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 09:15:58,900 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30865.88 MB 2025-02-14 09:15:58,900 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38889.59 MB 2025-02-14 09:15:58,900 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8023.70 MB 2025-02-14 09:15:58,900 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36418.13 MB 2025-02-14 09:15:59,067 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 09:15:59,067 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 09:15:59,067 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 09:15:59,067 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:15:59,067 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32407.39 MB 2025-02-14 09:15:59,067 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33174.39 MB 2025-02-14 09:15:59,067 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 09:15:59,067 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38889.59 MB 2025-02-14 09:15:59,067 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39306.92 MB 2025-02-14 09:15:59,067 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 09:15:59,067 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33882.18 MB 2025-02-14 09:15:59,086 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 09:15:59,086 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 09:15:59,086 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:15:59,086 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:15:59,086 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33587.28 MB 2025-02-14 09:15:59,086 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33815.75 MB 2025-02-14 09:15:59,086 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.47 MB 2025-02-14 09:15:59,086 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39306.92 MB 2025-02-14 09:15:59,086 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39306.92 MB 2025-02-14 09:15:59,086 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:15:59,086 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34035.68 MB 2025-02-14 09:15:59,087 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 09:15:59,088 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 09:15:59,088 - resource_logging.py:150 - __exit__ - DEBUG - Time: 33.47 seconds 2025-02-14 09:15:59,088 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:15:59,088 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19961.25 MB 2025-02-14 09:15:59,088 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34016.14 MB 2025-02-14 09:15:59,088 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14054.88 MB 2025-02-14 09:15:59,088 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41561.36 MB 2025-02-14 09:15:59,088 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39306.92 MB 2025-02-14 09:15:59,088 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2254.44 MB 2025-02-14 09:15:59,088 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34035.68 MB 2025-02-14 09:15:59,357 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 09:15:59,357 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 09:15:59,357 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 09:15:59,357 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:15:59,357 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34016.14 MB 2025-02-14 09:15:59,357 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24954.98 MB 2025-02-14 09:15:59,357 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9061.16 MB 2025-02-14 09:15:59,357 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39306.92 MB 2025-02-14 09:15:59,357 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39306.92 MB 2025-02-14 09:15:59,357 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:15:59,357 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36519.20 MB 2025-02-14 09:15:59,375 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8134, cut from 8136 2025-02-14 09:15:59,375 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 09:15:59,381 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 09:15:59,381 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 09:15:59,381 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:15:59,381 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:15:59,381 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24954.98 MB 2025-02-14 09:15:59,381 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33364.78 MB 2025-02-14 09:15:59,381 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8409.81 MB 2025-02-14 09:15:59,381 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39306.92 MB 2025-02-14 09:15:59,381 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47668.26 MB 2025-02-14 09:15:59,381 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8361.35 MB 2025-02-14 09:15:59,381 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33364.78 MB 2025-02-14 09:15:59,543 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7926] 2025-02-14 09:15:59,544 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:15:59,545 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 09:15:59,545 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:15:59,545 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 09:15:59,550 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 09:15:59,551 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:15:59,551 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 09:15:59,551 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 09:17:08,439 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:17:08,439 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 09:17:08,444 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 09:17:08,447 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:17:08,448 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 274, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 09:17:08,449 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:17:08,449 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 274, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 09:17:12,740 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 09:17:12,740 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 09:17:12,740 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.29 seconds 2025-02-14 09:17:12,740 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:17:12,740 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14877.98 MB 2025-02-14 09:17:12,740 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15847.65 MB 2025-02-14 09:17:12,740 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 969.67 MB 2025-02-14 09:17:12,740 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60209.23 MB 2025-02-14 09:17:12,740 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22034.78 MB 2025-02-14 09:17:12,740 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -38174.46 MB 2025-02-14 09:17:12,740 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24802.34 MB 2025-02-14 09:17:12,759 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 09:17:12,759 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 09:17:12,759 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:17:12,759 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:17:12,759 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15847.65 MB 2025-02-14 09:17:12,759 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16268.23 MB 2025-02-14 09:17:12,759 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 420.58 MB 2025-02-14 09:17:12,759 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22034.78 MB 2025-02-14 09:17:12,759 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22034.78 MB 2025-02-14 09:17:12,759 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:17:12,759 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19605.03 MB 2025-02-14 09:17:14,035 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 09:17:14,035 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 09:17:14,035 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.27 seconds 2025-02-14 09:17:14,035 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:17:14,035 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16268.23 MB 2025-02-14 09:17:14,035 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16622.57 MB 2025-02-14 09:17:14,035 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 354.34 MB 2025-02-14 09:17:14,035 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22034.78 MB 2025-02-14 09:17:14,035 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22034.78 MB 2025-02-14 09:17:14,035 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:17:14,035 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20608.54 MB 2025-02-14 09:17:14,045 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 09:17:14,045 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 09:17:14,045 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:17:14,045 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:17:14,045 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16622.57 MB 2025-02-14 09:17:14,045 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17883.57 MB 2025-02-14 09:17:14,045 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1261.00 MB 2025-02-14 09:17:14,045 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22034.78 MB 2025-02-14 09:17:14,045 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22034.78 MB 2025-02-14 09:17:14,045 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:17:14,045 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18829.71 MB 2025-02-14 09:17:14,189 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 09:17:14,189 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 09:17:14,189 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-14 09:17:14,189 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:17:14,189 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17883.57 MB 2025-02-14 09:17:14,189 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19380.62 MB 2025-02-14 09:17:14,189 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1497.05 MB 2025-02-14 09:17:14,189 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22034.78 MB 2025-02-14 09:17:14,189 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24561.84 MB 2025-02-14 09:17:14,189 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2527.07 MB 2025-02-14 09:17:14,189 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23081.41 MB 2025-02-14 09:17:14,190 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 09:17:14,190 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 09:17:14,190 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 09:17:14,190 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:17:14,190 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16622.57 MB 2025-02-14 09:17:14,190 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19380.62 MB 2025-02-14 09:17:14,190 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2758.05 MB 2025-02-14 09:17:14,190 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22034.78 MB 2025-02-14 09:17:14,190 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24561.84 MB 2025-02-14 09:17:14,190 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2527.07 MB 2025-02-14 09:17:14,190 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23081.41 MB 2025-02-14 09:17:14,306 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 09:17:14,306 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 09:17:14,306 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 09:17:14,306 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:17:14,306 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20404.26 MB 2025-02-14 09:17:14,306 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20916.23 MB 2025-02-14 09:17:14,306 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 511.97 MB 2025-02-14 09:17:14,306 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24561.84 MB 2025-02-14 09:17:14,307 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24836.57 MB 2025-02-14 09:17:14,307 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 274.73 MB 2025-02-14 09:17:14,307 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21388.68 MB 2025-02-14 09:17:14,321 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 09:17:14,321 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 09:17:14,321 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:17:14,321 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:17:14,321 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21191.84 MB 2025-02-14 09:17:14,321 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21400.84 MB 2025-02-14 09:17:14,321 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 209.00 MB 2025-02-14 09:17:14,321 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24836.57 MB 2025-02-14 09:17:14,321 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24838.67 MB 2025-02-14 09:17:14,321 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-14 09:17:14,321 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21499.58 MB 2025-02-14 09:17:14,322 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 09:17:14,322 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 09:17:14,322 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.87 seconds 2025-02-14 09:17:14,322 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:17:14,322 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13923.34 MB 2025-02-14 09:17:14,322 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21601.91 MB 2025-02-14 09:17:14,322 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7678.57 MB 2025-02-14 09:17:14,322 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60209.23 MB 2025-02-14 09:17:14,322 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24838.67 MB 2025-02-14 09:17:14,322 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35370.57 MB 2025-02-14 09:17:14,322 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21601.91 MB 2025-02-14 09:17:14,590 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 09:17:14,590 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 09:17:14,590 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 09:17:14,590 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:17:14,590 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21601.91 MB 2025-02-14 09:17:14,590 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24615.94 MB 2025-02-14 09:17:14,590 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-14 09:17:14,590 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24838.67 MB 2025-02-14 09:17:14,590 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26180.85 MB 2025-02-14 09:17:14,590 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1342.18 MB 2025-02-14 09:17:14,590 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24917.57 MB 2025-02-14 09:17:14,608 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 09:17:14,608 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 09:17:14,614 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 09:17:14,614 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 09:17:14,614 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:17:14,614 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:17:14,614 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18299.68 MB 2025-02-14 09:17:14,614 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26738.70 MB 2025-02-14 09:17:14,614 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 09:17:14,614 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26180.85 MB 2025-02-14 09:17:14,614 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34571.55 MB 2025-02-14 09:17:14,614 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 09:17:14,614 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26738.70 MB 2025-02-14 09:17:14,783 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 09:17:14,785 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:17:14,785 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 09:17:14,786 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:17:14,786 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 09:17:14,791 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 09:17:14,792 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:17:14,792 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 09:17:14,792 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 09:17:59,883 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:17:59,883 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 09:17:59,888 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 09:17:59,892 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:17:59,892 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1491, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 09:17:59,893 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:17:59,893 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1491, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 09:18:22,807 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 09:18:22,807 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 09:18:22,807 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.91 seconds 2025-02-14 09:18:22,807 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:18:22,807 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23358.23 MB 2025-02-14 09:18:22,807 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28634.80 MB 2025-02-14 09:18:22,807 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5276.57 MB 2025-02-14 09:18:22,807 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47156.56 MB 2025-02-14 09:18:22,807 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38950.40 MB 2025-02-14 09:18:22,807 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8206.16 MB 2025-02-14 09:18:22,807 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37586.70 MB 2025-02-14 09:18:22,890 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 09:18:22,890 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 09:18:22,891 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 09:18:22,891 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:18:22,891 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28634.80 MB 2025-02-14 09:18:22,891 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23529.10 MB 2025-02-14 09:18:22,891 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5105.70 MB 2025-02-14 09:18:22,891 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38950.40 MB 2025-02-14 09:18:22,891 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48272.24 MB 2025-02-14 09:18:22,891 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9321.84 MB 2025-02-14 09:18:22,891 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42662.70 MB 2025-02-14 09:18:24,850 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 09:18:24,850 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 09:18:24,850 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.96 seconds 2025-02-14 09:18:24,850 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:18:24,850 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23529.10 MB 2025-02-14 09:18:24,850 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24059.94 MB 2025-02-14 09:18:24,850 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 09:18:24,850 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48272.24 MB 2025-02-14 09:18:24,850 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29490.15 MB 2025-02-14 09:18:24,850 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18782.09 MB 2025-02-14 09:18:24,850 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28039.27 MB 2025-02-14 09:18:24,865 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 09:18:24,865 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 09:18:24,865 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:18:24,865 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:18:24,865 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24059.94 MB 2025-02-14 09:18:24,865 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25949.47 MB 2025-02-14 09:18:24,865 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 09:18:24,865 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29490.15 MB 2025-02-14 09:18:24,865 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30433.87 MB 2025-02-14 09:18:24,865 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 09:18:24,865 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27366.90 MB 2025-02-14 09:18:25,078 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 09:18:25,078 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 09:18:25,078 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 09:18:25,079 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:18:25,079 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25949.47 MB 2025-02-14 09:18:25,079 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28191.33 MB 2025-02-14 09:18:25,079 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 09:18:25,079 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30433.87 MB 2025-02-14 09:18:25,079 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36096.18 MB 2025-02-14 09:18:25,079 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 09:18:25,079 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33735.61 MB 2025-02-14 09:18:25,079 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 09:18:25,079 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 09:18:25,079 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 09:18:25,079 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:18:25,079 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24059.94 MB 2025-02-14 09:18:25,079 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28191.33 MB 2025-02-14 09:18:25,079 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 09:18:25,079 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29490.15 MB 2025-02-14 09:18:25,079 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36096.18 MB 2025-02-14 09:18:25,079 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 09:18:25,079 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33735.61 MB 2025-02-14 09:18:25,255 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 09:18:25,255 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 09:18:25,255 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 09:18:25,255 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:18:25,255 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29724.87 MB 2025-02-14 09:18:25,255 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30491.87 MB 2025-02-14 09:18:25,255 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 09:18:25,255 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36096.18 MB 2025-02-14 09:18:25,255 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36513.51 MB 2025-02-14 09:18:25,255 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 09:18:25,255 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31199.66 MB 2025-02-14 09:18:25,274 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 09:18:25,274 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 09:18:25,274 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:18:25,274 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:18:25,274 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30904.76 MB 2025-02-14 09:18:25,274 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31131.74 MB 2025-02-14 09:18:25,275 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.98 MB 2025-02-14 09:18:25,275 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36513.51 MB 2025-02-14 09:18:25,275 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36513.51 MB 2025-02-14 09:18:25,275 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:18:25,275 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31341.95 MB 2025-02-14 09:18:25,276 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 09:18:25,276 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 09:18:25,276 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.38 seconds 2025-02-14 09:18:25,276 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:18:25,276 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18163.47 MB 2025-02-14 09:18:25,276 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31331.68 MB 2025-02-14 09:18:25,276 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13168.21 MB 2025-02-14 09:18:25,276 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47156.56 MB 2025-02-14 09:18:25,276 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36513.51 MB 2025-02-14 09:18:25,276 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10643.05 MB 2025-02-14 09:18:25,276 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31341.95 MB 2025-02-14 09:18:25,543 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 09:18:25,543 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 09:18:25,543 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 09:18:25,543 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:18:25,543 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31331.68 MB 2025-02-14 09:18:25,543 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23150.34 MB 2025-02-14 09:18:25,543 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8181.34 MB 2025-02-14 09:18:25,543 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36513.51 MB 2025-02-14 09:18:25,543 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36513.51 MB 2025-02-14 09:18:25,543 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:18:25,543 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33829.22 MB 2025-02-14 09:18:25,561 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8116, cut from 8118 2025-02-14 09:18:25,561 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 09:18:25,567 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 09:18:25,568 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 09:18:25,568 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:18:25,568 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:18:25,568 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23150.34 MB 2025-02-14 09:18:25,568 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31542.76 MB 2025-02-14 09:18:25,568 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8392.42 MB 2025-02-14 09:18:25,568 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36513.51 MB 2025-02-14 09:18:25,568 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44855.98 MB 2025-02-14 09:18:25,568 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8342.47 MB 2025-02-14 09:18:25,568 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31542.76 MB 2025-02-14 09:18:25,737 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7908] 2025-02-14 09:18:25,738 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:18:25,738 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 09:18:25,739 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:18:25,739 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 09:18:25,744 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 09:18:25,745 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:18:25,745 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 09:18:25,745 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 09:19:14,375 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:19:14,375 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 09:19:14,380 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 09:19:14,384 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:19:14,384 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1198, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 09:19:14,385 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:19:14,385 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1198, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 09:19:32,673 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 09:19:32,673 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 09:19:32,673 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.28 seconds 2025-02-14 09:19:32,673 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:19:32,673 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21316.56 MB 2025-02-14 09:19:32,673 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25557.00 MB 2025-02-14 09:19:32,673 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4240.44 MB 2025-02-14 09:19:32,673 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53198.45 MB 2025-02-14 09:19:32,673 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33686.55 MB 2025-02-14 09:19:32,673 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19511.90 MB 2025-02-14 09:19:32,673 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34411.81 MB 2025-02-14 09:19:32,735 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 09:19:32,735 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 09:19:32,735 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 09:19:32,735 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:19:32,735 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25557.00 MB 2025-02-14 09:19:32,735 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22005.88 MB 2025-02-14 09:19:32,735 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3551.12 MB 2025-02-14 09:19:32,735 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33686.55 MB 2025-02-14 09:19:32,735 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39275.46 MB 2025-02-14 09:19:32,735 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5588.91 MB 2025-02-14 09:19:32,735 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35407.88 MB 2025-02-14 09:19:34,648 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 09:19:34,648 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 09:19:34,648 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 09:19:34,648 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:19:34,648 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22005.88 MB 2025-02-14 09:19:34,648 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22536.72 MB 2025-02-14 09:19:34,648 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 09:19:34,648 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39275.46 MB 2025-02-14 09:19:34,648 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26690.45 MB 2025-02-14 09:19:34,648 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12585.01 MB 2025-02-14 09:19:34,648 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26516.05 MB 2025-02-14 09:19:34,662 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 09:19:34,662 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 09:19:34,662 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:19:34,662 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:19:34,662 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22536.72 MB 2025-02-14 09:19:34,662 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24426.25 MB 2025-02-14 09:19:34,662 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 09:19:34,662 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26690.45 MB 2025-02-14 09:19:34,662 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28577.89 MB 2025-02-14 09:19:34,662 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 09:19:34,662 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25843.68 MB 2025-02-14 09:19:34,884 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 09:19:34,884 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 09:19:34,884 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 09:19:34,884 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:19:34,884 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24426.25 MB 2025-02-14 09:19:34,884 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26668.11 MB 2025-02-14 09:19:34,884 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 09:19:34,884 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28577.89 MB 2025-02-14 09:19:34,884 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34712.06 MB 2025-02-14 09:19:34,884 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 09:19:34,884 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32212.39 MB 2025-02-14 09:19:34,884 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 09:19:34,884 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 09:19:34,884 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 09:19:34,884 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:19:34,884 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22536.72 MB 2025-02-14 09:19:34,885 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26668.11 MB 2025-02-14 09:19:34,885 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 09:19:34,885 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26690.45 MB 2025-02-14 09:19:34,885 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34712.06 MB 2025-02-14 09:19:34,885 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-14 09:19:34,885 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32212.39 MB 2025-02-14 09:19:35,057 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 09:19:35,057 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 09:19:35,057 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 09:19:35,057 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:19:35,057 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28201.65 MB 2025-02-14 09:19:35,057 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28968.65 MB 2025-02-14 09:19:35,057 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 09:19:35,057 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34712.06 MB 2025-02-14 09:19:35,057 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35127.30 MB 2025-02-14 09:19:35,057 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 09:19:35,057 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29676.44 MB 2025-02-14 09:19:35,076 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 09:19:35,076 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 09:19:35,076 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:19:35,076 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:19:35,076 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29381.54 MB 2025-02-14 09:19:35,077 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29611.04 MB 2025-02-14 09:19:35,077 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.50 MB 2025-02-14 09:19:35,077 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35127.30 MB 2025-02-14 09:19:35,077 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35127.30 MB 2025-02-14 09:19:35,077 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:19:35,077 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29820.35 MB 2025-02-14 09:19:35,078 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 09:19:35,078 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 09:19:35,078 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.69 seconds 2025-02-14 09:19:35,078 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:19:35,078 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17142.63 MB 2025-02-14 09:19:35,078 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29811.85 MB 2025-02-14 09:19:35,078 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12669.21 MB 2025-02-14 09:19:35,078 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53198.45 MB 2025-02-14 09:19:35,078 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35127.30 MB 2025-02-14 09:19:35,078 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18071.16 MB 2025-02-14 09:19:35,078 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29820.35 MB 2025-02-14 09:19:35,345 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 09:19:35,345 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 09:19:35,345 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 09:19:35,345 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:19:35,345 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29811.85 MB 2025-02-14 09:19:35,345 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22143.37 MB 2025-02-14 09:19:35,345 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7668.47 MB 2025-02-14 09:19:35,345 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35127.30 MB 2025-02-14 09:19:35,345 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35127.30 MB 2025-02-14 09:19:35,345 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:19:35,345 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32320.14 MB 2025-02-14 09:19:35,363 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8151, cut from 8153 2025-02-14 09:19:35,363 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 09:19:35,369 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 09:19:35,369 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 09:19:35,369 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:19:35,369 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:19:35,369 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22143.37 MB 2025-02-14 09:19:35,369 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30570.71 MB 2025-02-14 09:19:35,369 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8427.34 MB 2025-02-14 09:19:35,369 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35127.30 MB 2025-02-14 09:19:35,369 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43507.52 MB 2025-02-14 09:19:35,369 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8380.22 MB 2025-02-14 09:19:35,369 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30570.71 MB 2025-02-14 09:19:35,533 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7943] 2025-02-14 09:19:35,535 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:19:35,535 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 09:19:35,536 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:19:35,536 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 09:19:35,540 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 09:19:35,541 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:19:35,541 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 09:19:35,542 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 09:21:26,371 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:21:26,371 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 09:21:26,376 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 09:21:26,379 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:21:26,379 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1074, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 09:21:26,380 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:21:26,380 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1074, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 09:21:42,742 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 09:21:42,742 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 09:21:42,742 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.36 seconds 2025-02-14 09:21:42,742 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:21:42,742 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20452.51 MB 2025-02-14 09:21:42,742 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24253.34 MB 2025-02-14 09:21:42,742 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3800.83 MB 2025-02-14 09:21:42,742 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51887.73 MB 2025-02-14 09:21:42,742 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29077.01 MB 2025-02-14 09:21:42,742 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22810.72 MB 2025-02-14 09:21:42,742 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33095.58 MB 2025-02-14 09:21:42,811 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 09:21:42,811 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 09:21:42,811 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 09:21:42,811 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:21:42,811 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24253.34 MB 2025-02-14 09:21:42,811 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21361.24 MB 2025-02-14 09:21:42,811 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2892.09 MB 2025-02-14 09:21:42,811 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29077.01 MB 2025-02-14 09:21:42,811 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42312.14 MB 2025-02-14 09:21:42,811 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 13235.13 MB 2025-02-14 09:21:42,811 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35916.22 MB 2025-02-14 09:21:44,706 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 09:21:44,706 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 09:21:44,706 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.89 seconds 2025-02-14 09:21:44,706 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:21:44,706 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21361.24 MB 2025-02-14 09:21:44,706 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21892.08 MB 2025-02-14 09:21:44,706 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 09:21:44,706 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42312.14 MB 2025-02-14 09:21:44,706 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26690.45 MB 2025-02-14 09:21:44,706 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15621.69 MB 2025-02-14 09:21:44,706 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25871.42 MB 2025-02-14 09:21:44,719 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 09:21:44,719 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 09:21:44,719 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:21:44,719 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:21:44,719 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21892.08 MB 2025-02-14 09:21:44,719 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23781.62 MB 2025-02-14 09:21:44,719 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 09:21:44,719 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26690.45 MB 2025-02-14 09:21:44,719 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28577.89 MB 2025-02-14 09:21:44,719 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 09:21:44,719 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25199.05 MB 2025-02-14 09:21:44,928 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 09:21:44,928 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 09:21:44,928 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 09:21:44,928 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:21:44,928 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23781.62 MB 2025-02-14 09:21:44,928 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26023.47 MB 2025-02-14 09:21:44,928 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 09:21:44,928 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28577.89 MB 2025-02-14 09:21:44,928 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34240.20 MB 2025-02-14 09:21:44,928 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 09:21:44,928 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31567.76 MB 2025-02-14 09:21:44,929 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 09:21:44,929 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 09:21:44,929 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 09:21:44,929 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:21:44,929 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21892.08 MB 2025-02-14 09:21:44,929 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26023.47 MB 2025-02-14 09:21:44,929 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 09:21:44,929 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26690.45 MB 2025-02-14 09:21:44,929 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34240.20 MB 2025-02-14 09:21:44,929 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-14 09:21:44,929 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31567.76 MB 2025-02-14 09:21:45,097 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 09:21:45,097 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 09:21:45,097 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 09:21:45,097 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:21:45,097 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27557.02 MB 2025-02-14 09:21:45,097 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28324.02 MB 2025-02-14 09:21:45,097 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 09:21:45,097 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34240.20 MB 2025-02-14 09:21:45,097 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34655.44 MB 2025-02-14 09:21:45,097 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 09:21:45,097 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29031.81 MB 2025-02-14 09:21:45,115 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 09:21:45,116 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 09:21:45,116 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:21:45,116 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:21:45,116 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28736.91 MB 2025-02-14 09:21:45,116 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28965.08 MB 2025-02-14 09:21:45,116 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.17 MB 2025-02-14 09:21:45,116 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34655.44 MB 2025-02-14 09:21:45,116 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34655.44 MB 2025-02-14 09:21:45,116 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:21:45,116 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29172.33 MB 2025-02-14 09:21:45,117 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 09:21:45,117 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 09:21:45,117 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.73 seconds 2025-02-14 09:21:45,117 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:21:45,117 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16710.61 MB 2025-02-14 09:21:45,117 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29165.34 MB 2025-02-14 09:21:45,117 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12454.73 MB 2025-02-14 09:21:45,117 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51887.73 MB 2025-02-14 09:21:45,117 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34655.44 MB 2025-02-14 09:21:45,117 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17232.30 MB 2025-02-14 09:21:45,117 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29172.33 MB 2025-02-14 09:21:45,381 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 09:21:45,381 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 09:21:45,381 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 09:21:45,381 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:21:45,381 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29165.34 MB 2025-02-14 09:21:45,381 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21702.43 MB 2025-02-14 09:21:45,381 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7462.92 MB 2025-02-14 09:21:45,381 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34655.44 MB 2025-02-14 09:21:45,381 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34655.44 MB 2025-02-14 09:21:45,381 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:21:45,381 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31666.87 MB 2025-02-14 09:21:45,399 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8129, cut from 8131 2025-02-14 09:21:45,399 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 09:21:45,405 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 09:21:45,405 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 09:21:45,405 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:21:45,405 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:21:45,405 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21702.43 MB 2025-02-14 09:21:45,405 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30107.54 MB 2025-02-14 09:21:45,405 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8405.11 MB 2025-02-14 09:21:45,405 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34655.44 MB 2025-02-14 09:21:45,405 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45103.45 MB 2025-02-14 09:21:45,405 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10448.01 MB 2025-02-14 09:21:45,405 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30107.54 MB 2025-02-14 09:21:45,569 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7921] 2025-02-14 09:21:45,570 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:21:45,570 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 09:21:45,571 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:21:45,571 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 09:21:45,576 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 09:21:45,577 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:21:45,577 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 09:21:45,577 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 09:23:02,355 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:23:02,355 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 09:23:02,360 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 09:23:02,364 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:23:02,364 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2518, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 09:23:02,366 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:23:02,366 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2518, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 09:23:41,073 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 09:23:41,073 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 09:23:41,073 - resource_logging.py:150 - __exit__ - DEBUG - Time: 38.69 seconds 2025-02-14 09:23:41,073 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:23:41,073 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30517.67 MB 2025-02-14 09:23:41,073 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39428.73 MB 2025-02-14 09:23:41,074 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8911.06 MB 2025-02-14 09:23:41,074 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 75187.09 MB 2025-02-14 09:23:41,074 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43371.20 MB 2025-02-14 09:23:41,074 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31815.89 MB 2025-02-14 09:23:41,074 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48370.07 MB 2025-02-14 09:23:41,240 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 09:23:41,240 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 09:23:41,240 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 09:23:41,241 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:23:41,241 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39428.73 MB 2025-02-14 09:23:41,241 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28870.37 MB 2025-02-14 09:23:41,241 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10558.36 MB 2025-02-14 09:23:41,241 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43371.20 MB 2025-02-14 09:23:41,241 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 77145.83 MB 2025-02-14 09:23:41,241 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 33774.63 MB 2025-02-14 09:23:41,241 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 65307.72 MB 2025-02-14 09:23:43,194 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 09:23:43,194 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 09:23:43,194 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.95 seconds 2025-02-14 09:23:43,194 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:23:43,194 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28870.37 MB 2025-02-14 09:23:43,194 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29401.21 MB 2025-02-14 09:23:43,194 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 09:23:43,194 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 77145.83 MB 2025-02-14 09:23:43,194 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31417.43 MB 2025-02-14 09:23:43,194 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -45728.40 MB 2025-02-14 09:23:43,194 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33381.58 MB 2025-02-14 09:23:43,208 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 09:23:43,208 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 09:23:43,208 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:23:43,208 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:23:43,208 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29401.21 MB 2025-02-14 09:23:43,208 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31290.75 MB 2025-02-14 09:23:43,208 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 09:23:43,208 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31417.43 MB 2025-02-14 09:23:43,208 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34720.45 MB 2025-02-14 09:23:43,208 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 09:23:43,208 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32708.17 MB 2025-02-14 09:23:43,418 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 09:23:43,418 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 09:23:43,418 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 09:23:43,418 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:23:43,418 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31290.75 MB 2025-02-14 09:23:43,418 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33532.60 MB 2025-02-14 09:23:43,418 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 09:23:43,418 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34720.45 MB 2025-02-14 09:23:43,418 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41326.48 MB 2025-02-14 09:23:43,418 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 09:23:43,418 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39076.88 MB 2025-02-14 09:23:43,419 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 09:23:43,419 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 09:23:43,419 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 09:23:43,419 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:23:43,419 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29401.21 MB 2025-02-14 09:23:43,419 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33532.60 MB 2025-02-14 09:23:43,419 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 09:23:43,419 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31417.43 MB 2025-02-14 09:23:43,419 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41326.48 MB 2025-02-14 09:23:43,419 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9909.04 MB 2025-02-14 09:23:43,419 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39076.88 MB 2025-02-14 09:23:43,587 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 09:23:43,587 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 09:23:43,587 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 09:23:43,587 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:23:43,587 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35066.14 MB 2025-02-14 09:23:43,588 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35833.15 MB 2025-02-14 09:23:43,588 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 09:23:43,588 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41326.48 MB 2025-02-14 09:23:43,588 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41741.71 MB 2025-02-14 09:23:43,588 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 09:23:43,588 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36540.93 MB 2025-02-14 09:23:43,607 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 09:23:43,607 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 09:23:43,607 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:23:43,607 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:23:43,607 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36246.03 MB 2025-02-14 09:23:43,607 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36475.29 MB 2025-02-14 09:23:43,607 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.26 MB 2025-02-14 09:23:43,607 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41741.71 MB 2025-02-14 09:23:43,607 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41741.71 MB 2025-02-14 09:23:43,607 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:23:43,607 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36688.84 MB 2025-02-14 09:23:43,608 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 09:23:43,608 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 09:23:43,608 - resource_logging.py:150 - __exit__ - DEBUG - Time: 41.24 seconds 2025-02-14 09:23:43,608 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:23:43,608 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21743.19 MB 2025-02-14 09:23:43,608 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36675.63 MB 2025-02-14 09:23:43,608 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14932.44 MB 2025-02-14 09:23:43,608 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 66412.61 MB 2025-02-14 09:23:43,608 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41741.71 MB 2025-02-14 09:23:43,608 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24670.90 MB 2025-02-14 09:23:43,608 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36688.84 MB 2025-02-14 09:23:43,877 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 09:23:43,877 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 09:23:43,877 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 09:23:43,877 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:23:43,877 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36675.63 MB 2025-02-14 09:23:43,877 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26736.15 MB 2025-02-14 09:23:43,877 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9939.47 MB 2025-02-14 09:23:43,877 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41741.71 MB 2025-02-14 09:23:43,877 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41741.71 MB 2025-02-14 09:23:43,877 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:23:43,877 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39178.08 MB 2025-02-14 09:23:43,895 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8132, cut from 8134 2025-02-14 09:23:43,895 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 09:23:43,901 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 09:23:43,901 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 09:23:43,901 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:23:43,901 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:23:43,901 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26736.15 MB 2025-02-14 09:23:43,901 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35144.41 MB 2025-02-14 09:23:43,901 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8408.25 MB 2025-02-14 09:23:43,901 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41741.71 MB 2025-02-14 09:23:43,901 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45921.34 MB 2025-02-14 09:23:43,901 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4179.62 MB 2025-02-14 09:23:43,901 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35144.41 MB 2025-02-14 09:23:44,064 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7924] 2025-02-14 09:23:44,065 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:23:44,065 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 09:23:44,066 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:23:44,066 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 09:23:44,071 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 09:23:44,072 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:23:44,072 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 09:23:44,072 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 09:24:21,061 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:24:21,062 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 09:24:21,066 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 09:24:21,070 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:24:21,070 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1901, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 09:24:21,071 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:24:21,071 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1901, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 09:24:50,576 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 09:24:50,576 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 09:24:50,576 - resource_logging.py:150 - __exit__ - DEBUG - Time: 29.50 seconds 2025-02-14 09:24:50,576 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:24:50,576 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26215.18 MB 2025-02-14 09:24:50,576 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32942.84 MB 2025-02-14 09:24:50,576 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6727.66 MB 2025-02-14 09:24:50,576 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54280.59 MB 2025-02-14 09:24:50,576 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40493.91 MB 2025-02-14 09:24:50,576 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13786.68 MB 2025-02-14 09:24:50,576 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41802.65 MB 2025-02-14 09:24:50,695 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 09:24:50,695 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 09:24:50,695 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 09:24:50,695 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:24:50,695 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32942.84 MB 2025-02-14 09:24:50,695 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25660.56 MB 2025-02-14 09:24:50,695 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7282.29 MB 2025-02-14 09:24:50,695 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40493.91 MB 2025-02-14 09:24:50,695 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 61020.83 MB 2025-02-14 09:24:50,695 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 20526.92 MB 2025-02-14 09:24:50,695 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51738.30 MB 2025-02-14 09:24:52,680 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 09:24:52,680 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 09:24:52,680 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.98 seconds 2025-02-14 09:24:52,680 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:24:52,680 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25660.56 MB 2025-02-14 09:24:52,680 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26191.40 MB 2025-02-14 09:24:52,680 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 09:24:52,680 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61020.83 MB 2025-02-14 09:24:52,680 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35181.82 MB 2025-02-14 09:24:52,680 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25839.01 MB 2025-02-14 09:24:52,680 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30170.73 MB 2025-02-14 09:24:52,693 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 09:24:52,693 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 09:24:52,693 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:24:52,693 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:24:52,693 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26191.40 MB 2025-02-14 09:24:52,693 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28080.93 MB 2025-02-14 09:24:52,693 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 09:24:52,693 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35181.82 MB 2025-02-14 09:24:52,693 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35181.82 MB 2025-02-14 09:24:52,693 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:24:52,693 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29498.36 MB 2025-02-14 09:24:52,909 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 09:24:52,909 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 09:24:52,909 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 09:24:52,909 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:24:52,909 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28080.93 MB 2025-02-14 09:24:52,909 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30322.79 MB 2025-02-14 09:24:52,909 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 09:24:52,909 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35181.82 MB 2025-02-14 09:24:52,909 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38956.70 MB 2025-02-14 09:24:52,909 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 09:24:52,909 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35867.07 MB 2025-02-14 09:24:52,910 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 09:24:52,910 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 09:24:52,910 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 09:24:52,910 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:24:52,910 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26191.40 MB 2025-02-14 09:24:52,910 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30322.79 MB 2025-02-14 09:24:52,910 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 09:24:52,910 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35181.82 MB 2025-02-14 09:24:52,910 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38956.70 MB 2025-02-14 09:24:52,910 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 09:24:52,910 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35867.07 MB 2025-02-14 09:24:53,086 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 09:24:53,086 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 09:24:53,086 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 09:24:53,086 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:24:53,086 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31856.33 MB 2025-02-14 09:24:53,086 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32623.33 MB 2025-02-14 09:24:53,086 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 09:24:53,086 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38956.70 MB 2025-02-14 09:24:53,086 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39371.93 MB 2025-02-14 09:24:53,086 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 09:24:53,086 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33331.12 MB 2025-02-14 09:24:53,106 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 09:24:53,106 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 09:24:53,106 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:24:53,106 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:24:53,106 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33036.22 MB 2025-02-14 09:24:53,106 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33264.52 MB 2025-02-14 09:24:53,106 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.30 MB 2025-02-14 09:24:53,106 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39371.93 MB 2025-02-14 09:24:53,106 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39371.93 MB 2025-02-14 09:24:53,106 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:24:53,106 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33487.66 MB 2025-02-14 09:24:53,107 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 09:24:53,107 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 09:24:53,107 - resource_logging.py:150 - __exit__ - DEBUG - Time: 32.03 seconds 2025-02-14 09:24:53,107 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:24:53,107 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19591.94 MB 2025-02-14 09:24:53,107 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33464.73 MB 2025-02-14 09:24:53,107 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13872.79 MB 2025-02-14 09:24:53,107 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54280.59 MB 2025-02-14 09:24:53,107 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39371.93 MB 2025-02-14 09:24:53,107 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14908.65 MB 2025-02-14 09:24:53,107 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33487.66 MB 2025-02-14 09:24:53,379 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 09:24:53,379 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 09:24:53,379 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 09:24:53,379 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:24:53,379 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33464.73 MB 2025-02-14 09:24:53,379 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24583.00 MB 2025-02-14 09:24:53,379 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8881.73 MB 2025-02-14 09:24:53,379 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39371.93 MB 2025-02-14 09:24:53,379 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39371.93 MB 2025-02-14 09:24:53,379 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:24:53,379 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35965.65 MB 2025-02-14 09:24:53,397 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8127, cut from 8129 2025-02-14 09:24:53,397 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 09:24:53,403 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 09:24:53,403 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 09:24:53,403 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:24:53,403 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:24:53,403 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24583.00 MB 2025-02-14 09:24:53,403 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32985.56 MB 2025-02-14 09:24:53,403 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8402.56 MB 2025-02-14 09:24:53,403 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39371.93 MB 2025-02-14 09:24:53,403 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43549.46 MB 2025-02-14 09:24:53,403 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4177.53 MB 2025-02-14 09:24:53,403 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32985.56 MB 2025-02-14 09:24:53,572 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7919] 2025-02-14 09:24:53,573 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:24:53,573 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 09:24:53,574 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:24:53,574 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 09:24:53,579 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 09:24:53,580 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:24:53,580 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 09:24:53,580 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 09:25:05,047 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:25:05,048 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 09:25:05,056 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 09:25:05,063 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:25:05,063 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 944, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 09:25:05,065 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:25:05,065 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 944, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 09:25:19,855 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 09:25:19,855 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 09:25:19,855 - resource_logging.py:150 - __exit__ - DEBUG - Time: 14.78 seconds 2025-02-14 09:25:19,855 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:25:19,855 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19546.65 MB 2025-02-14 09:25:19,855 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22887.41 MB 2025-02-14 09:25:19,855 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3340.76 MB 2025-02-14 09:25:19,855 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51904.51 MB 2025-02-14 09:25:19,855 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28607.25 MB 2025-02-14 09:25:19,855 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23297.26 MB 2025-02-14 09:25:19,855 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31736.74 MB 2025-02-14 09:25:19,907 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 09:25:19,907 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 09:25:19,907 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-14 09:25:19,907 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:25:19,907 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22887.41 MB 2025-02-14 09:25:19,907 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20685.41 MB 2025-02-14 09:25:19,907 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2202.00 MB 2025-02-14 09:25:19,907 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28607.25 MB 2025-02-14 09:25:19,907 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36574.33 MB 2025-02-14 09:25:19,907 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7967.08 MB 2025-02-14 09:25:19,907 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32305.53 MB 2025-02-14 09:25:21,831 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 09:25:21,831 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 09:25:21,831 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 09:25:21,831 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:25:21,831 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20685.41 MB 2025-02-14 09:25:21,831 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21216.25 MB 2025-02-14 09:25:21,831 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 09:25:21,831 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36574.33 MB 2025-02-14 09:25:21,831 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26682.06 MB 2025-02-14 09:25:21,831 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9892.27 MB 2025-02-14 09:25:21,831 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25195.59 MB 2025-02-14 09:25:21,845 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 09:25:21,845 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 09:25:21,845 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:25:21,845 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:25:21,845 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21216.25 MB 2025-02-14 09:25:21,845 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23105.79 MB 2025-02-14 09:25:21,845 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 09:25:21,845 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26682.06 MB 2025-02-14 09:25:21,845 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27625.78 MB 2025-02-14 09:25:21,845 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 09:25:21,845 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24523.22 MB 2025-02-14 09:25:22,057 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 09:25:22,057 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 09:25:22,057 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 09:25:22,057 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:25:22,057 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23105.79 MB 2025-02-14 09:25:22,057 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25347.64 MB 2025-02-14 09:25:22,057 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 09:25:22,057 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27625.78 MB 2025-02-14 09:25:22,057 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33759.95 MB 2025-02-14 09:25:22,057 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 09:25:22,057 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30891.93 MB 2025-02-14 09:25:22,058 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 09:25:22,058 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 09:25:22,058 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 09:25:22,058 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:25:22,058 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21216.25 MB 2025-02-14 09:25:22,058 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25347.64 MB 2025-02-14 09:25:22,058 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 09:25:22,058 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26682.06 MB 2025-02-14 09:25:22,058 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33759.95 MB 2025-02-14 09:25:22,058 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7077.89 MB 2025-02-14 09:25:22,058 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30891.93 MB 2025-02-14 09:25:22,226 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 09:25:22,226 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 09:25:22,226 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 09:25:22,226 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:25:22,226 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26881.19 MB 2025-02-14 09:25:22,226 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27648.19 MB 2025-02-14 09:25:22,226 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 09:25:22,226 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33759.95 MB 2025-02-14 09:25:22,226 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34175.19 MB 2025-02-14 09:25:22,226 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 09:25:22,226 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28355.98 MB 2025-02-14 09:25:22,246 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 09:25:22,246 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 09:25:22,246 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:25:22,246 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:25:22,246 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28061.08 MB 2025-02-14 09:25:22,246 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28290.25 MB 2025-02-14 09:25:22,246 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.17 MB 2025-02-14 09:25:22,246 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34175.19 MB 2025-02-14 09:25:22,246 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34175.19 MB 2025-02-14 09:25:22,246 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:25:22,246 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28497.53 MB 2025-02-14 09:25:22,247 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 09:25:22,247 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 09:25:22,247 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.18 seconds 2025-02-14 09:25:22,247 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:25:22,247 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16257.68 MB 2025-02-14 09:25:22,247 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28491.32 MB 2025-02-14 09:25:22,247 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12233.64 MB 2025-02-14 09:25:22,247 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51904.51 MB 2025-02-14 09:25:22,247 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34175.19 MB 2025-02-14 09:25:22,247 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17729.32 MB 2025-02-14 09:25:22,247 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28497.53 MB 2025-02-14 09:25:22,518 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 09:25:22,518 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 09:25:22,518 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 09:25:22,518 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:25:22,518 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28491.32 MB 2025-02-14 09:25:22,518 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21262.07 MB 2025-02-14 09:25:22,518 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7229.25 MB 2025-02-14 09:25:22,518 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34175.19 MB 2025-02-14 09:25:22,518 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34175.19 MB 2025-02-14 09:25:22,518 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:25:22,518 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31002.99 MB 2025-02-14 09:25:22,544 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 09:25:22,545 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 09:25:22,598 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 09:25:22,598 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 09:25:22,598 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 09:25:22,598 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:25:22,598 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21262.07 MB 2025-02-14 09:25:22,598 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29701.09 MB 2025-02-14 09:25:22,598 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 09:25:22,598 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34175.19 MB 2025-02-14 09:25:22,598 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42565.89 MB 2025-02-14 09:25:22,598 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 09:25:22,598 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29701.09 MB 2025-02-14 09:25:22,767 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 09:25:22,768 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:25:22,768 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 09:25:22,769 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:25:22,769 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 09:25:22,774 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 09:25:22,775 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:25:22,775 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 09:25:22,775 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 09:25:34,869 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:25:34,869 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 09:25:34,874 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 09:25:34,877 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:25:34,877 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 184, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 09:25:34,878 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:25:34,878 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 184, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 09:25:37,753 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 09:25:37,753 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 09:25:37,754 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.87 seconds 2025-02-14 09:25:37,754 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:25:37,754 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14250.85 MB 2025-02-14 09:25:37,754 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14902.01 MB 2025-02-14 09:25:37,754 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 651.17 MB 2025-02-14 09:25:37,754 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55150.90 MB 2025-02-14 09:25:37,754 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 16909.34 MB 2025-02-14 09:25:37,754 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -38241.57 MB 2025-02-14 09:25:37,754 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23723.03 MB 2025-02-14 09:25:37,766 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 09:25:37,766 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 09:25:37,766 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:25:37,766 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:25:37,766 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14902.01 MB 2025-02-14 09:25:37,767 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15148.06 MB 2025-02-14 09:25:37,767 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 246.04 MB 2025-02-14 09:25:37,767 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 16909.34 MB 2025-02-14 09:25:37,767 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18442.35 MB 2025-02-14 09:25:37,767 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1533.02 MB 2025-02-14 09:25:37,767 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17375.98 MB 2025-02-14 09:25:38,616 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 09:25:38,616 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 09:25:38,616 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.85 seconds 2025-02-14 09:25:38,616 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:25:38,616 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15148.06 MB 2025-02-14 09:25:38,616 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15378.97 MB 2025-02-14 09:25:38,616 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 230.92 MB 2025-02-14 09:25:38,616 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18442.35 MB 2025-02-14 09:25:38,616 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17815.31 MB 2025-02-14 09:25:38,616 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -627.05 MB 2025-02-14 09:25:38,616 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19319.53 MB 2025-02-14 09:25:38,624 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 09:25:38,624 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 09:25:38,624 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:25:38,624 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:25:38,624 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15378.91 MB 2025-02-14 09:25:38,624 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16201.18 MB 2025-02-14 09:25:38,624 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 822.27 MB 2025-02-14 09:25:38,624 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17815.31 MB 2025-02-14 09:25:38,624 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18226.35 MB 2025-02-14 09:25:38,624 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 411.04 MB 2025-02-14 09:25:38,624 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16817.77 MB 2025-02-14 09:25:38,719 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 09:25:38,719 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 09:25:38,719 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 09:25:38,719 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:25:38,719 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16201.18 MB 2025-02-14 09:25:38,719 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17176.42 MB 2025-02-14 09:25:38,719 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 975.24 MB 2025-02-14 09:25:38,719 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18226.35 MB 2025-02-14 09:25:38,719 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20487.08 MB 2025-02-14 09:25:38,719 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2260.73 MB 2025-02-14 09:25:38,719 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19590.51 MB 2025-02-14 09:25:38,720 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 09:25:38,720 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 09:25:38,720 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 09:25:38,720 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:25:38,720 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15378.91 MB 2025-02-14 09:25:38,720 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17176.42 MB 2025-02-14 09:25:38,720 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1797.52 MB 2025-02-14 09:25:38,720 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17815.31 MB 2025-02-14 09:25:38,720 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20487.08 MB 2025-02-14 09:25:38,720 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2671.77 MB 2025-02-14 09:25:38,720 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19590.51 MB 2025-02-14 09:25:38,796 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 09:25:38,796 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 09:25:38,796 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 09:25:38,796 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:25:38,796 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17843.52 MB 2025-02-14 09:25:38,796 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18177.42 MB 2025-02-14 09:25:38,796 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 333.91 MB 2025-02-14 09:25:38,796 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20487.08 MB 2025-02-14 09:25:38,796 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20665.34 MB 2025-02-14 09:25:38,796 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 178.26 MB 2025-02-14 09:25:38,796 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18491.38 MB 2025-02-14 09:25:38,806 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 09:25:38,806 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 09:25:38,806 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:25:38,806 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:25:38,806 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18357.04 MB 2025-02-14 09:25:38,806 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18577.01 MB 2025-02-14 09:25:38,806 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 219.98 MB 2025-02-14 09:25:38,806 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20665.34 MB 2025-02-14 09:25:38,806 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20665.34 MB 2025-02-14 09:25:38,806 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:25:38,806 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18602.65 MB 2025-02-14 09:25:38,807 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 09:25:38,807 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 09:25:38,807 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.93 seconds 2025-02-14 09:25:38,807 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:25:38,807 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13609.78 MB 2025-02-14 09:25:38,807 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18777.74 MB 2025-02-14 09:25:38,807 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5167.97 MB 2025-02-14 09:25:38,807 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55150.90 MB 2025-02-14 09:25:38,807 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20665.34 MB 2025-02-14 09:25:38,807 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34485.57 MB 2025-02-14 09:25:38,807 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18777.74 MB 2025-02-14 09:25:39,076 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 09:25:39,076 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 09:25:39,076 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 09:25:39,076 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:25:39,076 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18777.74 MB 2025-02-14 09:25:39,076 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17542.53 MB 2025-02-14 09:25:39,076 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1235.21 MB 2025-02-14 09:25:39,076 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20665.34 MB 2025-02-14 09:25:39,076 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20665.34 MB 2025-02-14 09:25:39,076 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:25:39,076 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19011.76 MB 2025-02-14 09:25:39,094 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8148, cut from 8150 2025-02-14 09:25:39,094 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 09:25:39,100 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 09:25:39,100 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 09:25:39,100 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:25:39,100 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:25:39,100 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17542.53 MB 2025-02-14 09:25:39,100 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25967.48 MB 2025-02-14 09:25:39,100 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8424.95 MB 2025-02-14 09:25:39,100 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20665.34 MB 2025-02-14 09:25:39,100 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31136.42 MB 2025-02-14 09:25:39,100 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10471.08 MB 2025-02-14 09:25:39,100 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25967.48 MB 2025-02-14 09:25:39,266 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7940] 2025-02-14 09:25:39,267 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:25:39,267 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 09:25:39,268 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:25:39,268 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 09:25:39,273 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 09:25:39,274 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:25:39,274 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 09:25:39,274 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 09:26:24,561 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:26:24,561 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 09:26:24,568 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 09:26:24,574 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:26:24,574 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 246, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 09:26:24,576 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:26:24,576 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 246, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 09:26:28,435 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 09:26:28,435 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 09:26:28,435 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.85 seconds 2025-02-14 09:26:28,435 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:26:28,435 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14682.87 MB 2025-02-14 09:26:28,435 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15553.45 MB 2025-02-14 09:26:28,435 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 870.58 MB 2025-02-14 09:26:28,435 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39512.44 MB 2025-02-14 09:26:28,435 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17523.80 MB 2025-02-14 09:26:28,435 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21988.64 MB 2025-02-14 09:26:28,435 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24381.54 MB 2025-02-14 09:26:28,450 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 09:26:28,450 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 09:26:28,450 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:26:28,450 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:26:28,450 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15553.45 MB 2025-02-14 09:26:28,450 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14830.50 MB 2025-02-14 09:26:28,450 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -722.96 MB 2025-02-14 09:26:28,450 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17523.80 MB 2025-02-14 09:26:28,450 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17523.80 MB 2025-02-14 09:26:28,450 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:26:28,450 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16740.58 MB 2025-02-14 09:26:28,875 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 09:26:28,875 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 09:26:28,875 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.42 seconds 2025-02-14 09:26:28,875 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:26:28,875 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14830.50 MB 2025-02-14 09:26:28,875 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14940.64 MB 2025-02-14 09:26:28,875 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 110.15 MB 2025-02-14 09:26:28,875 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17523.80 MB 2025-02-14 09:26:28,875 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17926.46 MB 2025-02-14 09:26:28,875 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 402.65 MB 2025-02-14 09:26:28,875 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18917.04 MB 2025-02-14 09:26:28,883 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 09:26:28,884 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 09:26:28,884 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 09:26:28,884 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:26:28,884 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14940.58 MB 2025-02-14 09:26:28,884 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15332.56 MB 2025-02-14 09:26:28,884 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 391.98 MB 2025-02-14 09:26:28,884 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17926.46 MB 2025-02-14 09:26:28,884 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17926.46 MB 2025-02-14 09:26:28,884 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:26:28,884 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15626.69 MB 2025-02-14 09:26:28,982 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 09:26:28,982 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 09:26:28,982 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 09:26:28,982 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:26:28,982 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15332.56 MB 2025-02-14 09:26:28,983 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15808.68 MB 2025-02-14 09:26:28,983 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 476.12 MB 2025-02-14 09:26:28,983 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17926.46 MB 2025-02-14 09:26:28,983 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17926.46 MB 2025-02-14 09:26:28,983 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:26:28,983 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16948.19 MB 2025-02-14 09:26:28,984 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 09:26:28,984 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 09:26:28,984 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 09:26:28,984 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:26:28,984 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14940.58 MB 2025-02-14 09:26:28,984 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15808.68 MB 2025-02-14 09:26:28,984 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 868.10 MB 2025-02-14 09:26:28,984 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17926.46 MB 2025-02-14 09:26:28,984 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17926.46 MB 2025-02-14 09:26:28,984 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:26:28,984 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16948.19 MB 2025-02-14 09:26:29,049 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 09:26:29,049 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 09:26:29,049 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 09:26:29,049 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:26:29,049 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16268.94 MB 2025-02-14 09:26:29,049 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16468.89 MB 2025-02-14 09:26:29,049 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 199.95 MB 2025-02-14 09:26:29,049 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17926.46 MB 2025-02-14 09:26:29,049 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18054.38 MB 2025-02-14 09:26:29,049 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 127.93 MB 2025-02-14 09:26:29,049 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16615.75 MB 2025-02-14 09:26:29,057 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 09:26:29,057 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 09:26:29,058 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 09:26:29,058 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:26:29,058 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16595.37 MB 2025-02-14 09:26:29,058 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16794.25 MB 2025-02-14 09:26:29,058 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 198.88 MB 2025-02-14 09:26:29,058 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18054.38 MB 2025-02-14 09:26:29,058 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18054.38 MB 2025-02-14 09:26:29,058 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:26:29,058 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16794.25 MB 2025-02-14 09:26:29,060 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 09:26:29,060 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 09:26:29,060 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.48 seconds 2025-02-14 09:26:29,060 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:26:29,060 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13825.79 MB 2025-02-14 09:26:29,060 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16971.54 MB 2025-02-14 09:26:29,060 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3145.75 MB 2025-02-14 09:26:29,060 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39512.44 MB 2025-02-14 09:26:29,060 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18054.38 MB 2025-02-14 09:26:29,060 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21458.06 MB 2025-02-14 09:26:29,060 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16971.54 MB 2025-02-14 09:26:29,309 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 09:26:29,310 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 09:26:29,310 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.25 seconds 2025-02-14 09:26:29,310 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:26:29,310 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14308.19 MB 2025-02-14 09:26:29,310 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16965.74 MB 2025-02-14 09:26:29,310 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2657.55 MB 2025-02-14 09:26:29,310 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18054.38 MB 2025-02-14 09:26:29,310 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18054.38 MB 2025-02-14 09:26:29,310 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:26:29,310 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17231.47 MB 2025-02-14 09:26:29,327 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 7195, cut from 7197 2025-02-14 09:26:29,327 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 09:26:29,334 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 09:26:29,334 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 09:26:29,334 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:26:29,334 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:26:29,334 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16965.74 MB 2025-02-14 09:26:29,334 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24407.29 MB 2025-02-14 09:26:29,334 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7441.55 MB 2025-02-14 09:26:29,334 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18054.38 MB 2025-02-14 09:26:29,334 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27302.82 MB 2025-02-14 09:26:29,334 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9248.44 MB 2025-02-14 09:26:29,334 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24407.29 MB 2025-02-14 09:26:29,534 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 6987] 2025-02-14 09:26:29,536 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:26:29,536 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 09:26:29,538 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:26:29,538 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 09:26:29,545 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 09:26:29,547 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:26:29,547 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 09:26:29,547 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 09:27:04,712 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:27:04,712 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 09:27:04,717 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 09:27:04,720 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:27:04,721 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1104, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 09:27:04,721 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:27:04,722 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1104, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 09:27:21,628 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 09:27:21,629 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 09:27:21,629 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.90 seconds 2025-02-14 09:27:21,629 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:27:21,629 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20661.56 MB 2025-02-14 09:27:21,629 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24568.55 MB 2025-02-14 09:27:21,629 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3906.99 MB 2025-02-14 09:27:21,629 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34701.57 MB 2025-02-14 09:27:21,629 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35611.74 MB 2025-02-14 09:27:21,629 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 910.16 MB 2025-02-14 09:27:21,629 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33530.31 MB 2025-02-14 09:27:21,669 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 09:27:21,669 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 09:27:21,670 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-14 09:27:21,670 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:27:21,670 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24568.55 MB 2025-02-14 09:27:21,670 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20330.31 MB 2025-02-14 09:27:21,670 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4238.24 MB 2025-02-14 09:27:21,670 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35611.74 MB 2025-02-14 09:27:21,670 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35611.74 MB 2025-02-14 09:27:21,670 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:27:21,670 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27870.60 MB 2025-02-14 09:27:22,774 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 09:27:22,774 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 09:27:22,774 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.10 seconds 2025-02-14 09:27:22,774 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:27:22,774 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20330.31 MB 2025-02-14 09:27:22,774 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20636.87 MB 2025-02-14 09:27:22,774 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 306.56 MB 2025-02-14 09:27:22,774 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35611.74 MB 2025-02-14 09:27:22,774 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28005.37 MB 2025-02-14 09:27:22,774 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7606.37 MB 2025-02-14 09:27:22,774 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24585.68 MB 2025-02-14 09:27:22,783 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 09:27:22,783 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 09:27:22,783 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:27:22,783 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:27:22,783 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20636.87 MB 2025-02-14 09:27:22,783 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21727.81 MB 2025-02-14 09:27:22,783 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1090.94 MB 2025-02-14 09:27:22,783 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28005.37 MB 2025-02-14 09:27:22,783 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28005.37 MB 2025-02-14 09:27:22,783 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:27:22,783 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22546.38 MB 2025-02-14 09:27:22,906 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 09:27:22,906 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 09:27:22,906 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 09:27:22,906 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:27:22,906 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21727.81 MB 2025-02-14 09:27:22,906 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23022.51 MB 2025-02-14 09:27:22,906 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1294.70 MB 2025-02-14 09:27:22,906 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28005.37 MB 2025-02-14 09:27:22,906 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28005.37 MB 2025-02-14 09:27:22,906 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:27:22,906 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26224.31 MB 2025-02-14 09:27:22,906 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 09:27:22,906 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 09:27:22,906 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 09:27:22,906 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:27:22,906 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20636.87 MB 2025-02-14 09:27:22,906 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23022.51 MB 2025-02-14 09:27:22,906 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2385.64 MB 2025-02-14 09:27:22,906 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28005.37 MB 2025-02-14 09:27:22,907 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28005.37 MB 2025-02-14 09:27:22,907 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:27:22,907 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26224.31 MB 2025-02-14 09:27:23,007 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 09:27:23,007 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 09:27:23,007 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 09:27:23,007 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:27:23,007 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23908.13 MB 2025-02-14 09:27:23,007 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24351.08 MB 2025-02-14 09:27:23,007 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 442.94 MB 2025-02-14 09:27:23,007 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28005.37 MB 2025-02-14 09:27:23,007 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28246.54 MB 2025-02-14 09:27:23,007 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 241.17 MB 2025-02-14 09:27:23,007 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24759.82 MB 2025-02-14 09:27:23,020 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 09:27:23,020 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 09:27:23,020 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:27:23,020 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:27:23,020 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24589.52 MB 2025-02-14 09:27:23,020 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24811.70 MB 2025-02-14 09:27:23,020 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 222.17 MB 2025-02-14 09:27:23,020 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28246.54 MB 2025-02-14 09:27:23,020 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28246.54 MB 2025-02-14 09:27:23,020 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:27:23,020 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24882.49 MB 2025-02-14 09:27:23,021 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 09:27:23,021 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 09:27:23,021 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.30 seconds 2025-02-14 09:27:23,021 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:27:23,021 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16815.13 MB 2025-02-14 09:27:23,021 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25012.77 MB 2025-02-14 09:27:23,021 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8197.64 MB 2025-02-14 09:27:23,021 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34701.57 MB 2025-02-14 09:27:23,021 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28246.54 MB 2025-02-14 09:27:23,021 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6455.03 MB 2025-02-14 09:27:23,021 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25012.77 MB 2025-02-14 09:27:23,287 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 09:27:23,287 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 09:27:23,287 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 09:27:23,287 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:27:23,287 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18007.93 MB 2025-02-14 09:27:23,287 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21021.96 MB 2025-02-14 09:27:23,287 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-14 09:27:23,287 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28246.54 MB 2025-02-14 09:27:23,287 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28246.54 MB 2025-02-14 09:27:23,287 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:27:23,287 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21323.33 MB 2025-02-14 09:27:23,305 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 09:27:23,305 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 09:27:23,311 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 09:27:23,311 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 09:27:23,311 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:27:23,311 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:27:23,311 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21021.96 MB 2025-02-14 09:27:23,311 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29460.98 MB 2025-02-14 09:27:23,311 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 09:27:23,311 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28246.54 MB 2025-02-14 09:27:23,311 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36637.25 MB 2025-02-14 09:27:23,311 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 09:27:23,311 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29460.98 MB 2025-02-14 09:27:23,476 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 09:27:23,477 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:27:23,477 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 09:27:23,478 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:27:23,478 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 09:27:23,483 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 09:27:23,485 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:27:23,485 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 09:27:23,485 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 09:27:38,588 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:27:38,588 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 09:27:38,593 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 09:27:38,597 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:27:38,597 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 712, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 09:27:38,598 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:27:38,598 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 712, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 09:27:49,626 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 09:27:49,626 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 09:27:49,626 - resource_logging.py:150 - __exit__ - DEBUG - Time: 11.02 seconds 2025-02-14 09:27:49,626 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:27:49,626 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17930.04 MB 2025-02-14 09:27:49,626 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20450.81 MB 2025-02-14 09:27:49,626 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2520.78 MB 2025-02-14 09:27:49,626 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49222.25 MB 2025-02-14 09:27:49,626 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26826.77 MB 2025-02-14 09:27:49,626 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22395.49 MB 2025-02-14 09:27:49,626 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29439.84 MB 2025-02-14 09:27:49,670 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 09:27:49,670 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 09:27:49,670 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-14 09:27:49,670 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:27:49,670 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20450.81 MB 2025-02-14 09:27:49,670 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19479.32 MB 2025-02-14 09:27:49,670 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -971.50 MB 2025-02-14 09:27:49,670 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26826.77 MB 2025-02-14 09:27:49,670 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33101.45 MB 2025-02-14 09:27:49,670 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6274.68 MB 2025-02-14 09:27:49,670 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29305.00 MB 2025-02-14 09:27:51,583 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 09:27:51,583 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 09:27:51,583 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 09:27:51,583 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:27:51,583 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19479.32 MB 2025-02-14 09:27:51,583 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20010.16 MB 2025-02-14 09:27:51,583 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 09:27:51,583 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33101.45 MB 2025-02-14 09:27:51,583 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25721.57 MB 2025-02-14 09:27:51,583 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7379.88 MB 2025-02-14 09:27:51,583 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23989.49 MB 2025-02-14 09:27:51,596 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 09:27:51,596 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 09:27:51,596 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:27:51,596 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:27:51,596 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20010.16 MB 2025-02-14 09:27:51,597 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21899.69 MB 2025-02-14 09:27:51,597 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 09:27:51,597 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25721.57 MB 2025-02-14 09:27:51,597 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25721.57 MB 2025-02-14 09:27:51,597 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:27:51,597 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23317.12 MB 2025-02-14 09:27:51,945 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 09:27:51,945 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 09:27:51,945 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.35 seconds 2025-02-14 09:27:51,945 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:27:51,945 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21899.69 MB 2025-02-14 09:27:51,945 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24141.55 MB 2025-02-14 09:27:51,945 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 09:27:51,945 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25721.57 MB 2025-02-14 09:27:51,945 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31855.74 MB 2025-02-14 09:27:51,945 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 09:27:51,945 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29685.83 MB 2025-02-14 09:27:51,946 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 09:27:51,946 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 09:27:51,946 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.36 seconds 2025-02-14 09:27:51,946 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:27:51,946 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20010.16 MB 2025-02-14 09:27:51,946 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24141.55 MB 2025-02-14 09:27:51,946 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 09:27:51,946 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25721.57 MB 2025-02-14 09:27:51,946 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31855.74 MB 2025-02-14 09:27:51,946 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 09:27:51,946 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29685.83 MB 2025-02-14 09:27:52,114 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 09:27:52,114 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 09:27:52,114 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 09:27:52,114 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:27:52,114 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25675.09 MB 2025-02-14 09:27:52,114 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26442.09 MB 2025-02-14 09:27:52,114 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 09:27:52,114 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31855.74 MB 2025-02-14 09:27:52,114 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32273.07 MB 2025-02-14 09:27:52,114 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 09:27:52,114 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27149.88 MB 2025-02-14 09:27:52,133 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 09:27:52,133 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 09:27:52,133 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:27:52,133 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:27:52,133 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26854.98 MB 2025-02-14 09:27:52,133 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27083.65 MB 2025-02-14 09:27:52,133 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.67 MB 2025-02-14 09:27:52,133 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32273.07 MB 2025-02-14 09:27:52,133 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32273.07 MB 2025-02-14 09:27:52,133 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:27:52,133 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27269.98 MB 2025-02-14 09:27:52,134 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 09:27:52,134 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 09:27:52,134 - resource_logging.py:150 - __exit__ - DEBUG - Time: 13.53 seconds 2025-02-14 09:27:52,134 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:27:52,134 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15449.37 MB 2025-02-14 09:27:52,134 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27284.72 MB 2025-02-14 09:27:52,134 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11835.35 MB 2025-02-14 09:27:52,134 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49222.25 MB 2025-02-14 09:27:52,134 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32273.07 MB 2025-02-14 09:27:52,134 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16949.18 MB 2025-02-14 09:27:52,134 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27284.72 MB 2025-02-14 09:27:52,403 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 09:27:52,403 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 09:27:52,403 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 09:27:52,403 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:27:52,403 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27284.72 MB 2025-02-14 09:27:52,403 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20453.76 MB 2025-02-14 09:27:52,403 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6830.96 MB 2025-02-14 09:27:52,403 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32273.07 MB 2025-02-14 09:27:52,403 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32273.07 MB 2025-02-14 09:27:52,403 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:27:52,403 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27284.72 MB 2025-02-14 09:27:52,421 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 09:27:52,421 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 09:27:52,427 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 09:27:52,427 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 09:27:52,427 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:27:52,428 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:27:52,428 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20453.76 MB 2025-02-14 09:27:52,428 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28892.78 MB 2025-02-14 09:27:52,428 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 09:27:52,428 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32273.07 MB 2025-02-14 09:27:52,428 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40663.78 MB 2025-02-14 09:27:52,428 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 09:27:52,428 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28892.78 MB 2025-02-14 09:27:52,590 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 09:27:52,592 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:27:52,592 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 09:27:52,593 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:27:52,593 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 09:27:52,597 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 09:27:52,598 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:27:52,599 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 09:27:52,599 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 09:28:08,439 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:28:08,439 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 09:28:08,446 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 09:28:08,451 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:28:08,452 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 294, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 09:28:08,453 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:28:08,453 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 294, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 09:28:13,113 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 09:28:13,113 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 09:28:13,113 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.65 seconds 2025-02-14 09:28:13,113 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:28:13,113 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23081.49 MB 2025-02-14 09:28:13,113 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24121.94 MB 2025-02-14 09:28:13,113 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1040.45 MB 2025-02-14 09:28:13,113 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53248.79 MB 2025-02-14 09:28:13,113 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27734.84 MB 2025-02-14 09:28:13,113 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25513.95 MB 2025-02-14 09:28:13,113 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33006.65 MB 2025-02-14 09:28:13,131 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 09:28:13,131 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 09:28:13,131 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:28:13,131 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:28:13,131 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24121.94 MB 2025-02-14 09:28:13,131 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23024.79 MB 2025-02-14 09:28:13,131 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1097.15 MB 2025-02-14 09:28:13,131 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27734.84 MB 2025-02-14 09:28:13,131 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27734.84 MB 2025-02-14 09:28:13,131 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:28:13,131 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25070.28 MB 2025-02-14 09:28:13,483 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 09:28:13,483 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 09:28:13,483 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.35 seconds 2025-02-14 09:28:13,483 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:28:13,483 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23024.79 MB 2025-02-14 09:28:13,483 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23112.31 MB 2025-02-14 09:28:13,483 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 87.52 MB 2025-02-14 09:28:13,483 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27734.84 MB 2025-02-14 09:28:13,483 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26692.55 MB 2025-02-14 09:28:13,483 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1042.28 MB 2025-02-14 09:28:13,483 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27111.26 MB 2025-02-14 09:28:13,491 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 09:28:13,491 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 09:28:13,491 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 09:28:13,491 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:28:13,491 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23112.31 MB 2025-02-14 09:28:13,491 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23424.01 MB 2025-02-14 09:28:13,491 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 311.70 MB 2025-02-14 09:28:13,491 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26692.55 MB 2025-02-14 09:28:13,491 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26692.55 MB 2025-02-14 09:28:13,491 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:28:13,491 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23657.89 MB 2025-02-14 09:28:13,571 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 09:28:13,571 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 09:28:13,572 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 09:28:13,572 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:28:13,572 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23424.01 MB 2025-02-14 09:28:13,572 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23802.62 MB 2025-02-14 09:28:13,572 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 378.61 MB 2025-02-14 09:28:13,572 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26692.55 MB 2025-02-14 09:28:13,572 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26692.55 MB 2025-02-14 09:28:13,572 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:28:13,572 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24708.72 MB 2025-02-14 09:28:13,573 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 09:28:13,573 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 09:28:13,573 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 09:28:13,573 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:28:13,573 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23112.31 MB 2025-02-14 09:28:13,573 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23802.62 MB 2025-02-14 09:28:13,573 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 690.31 MB 2025-02-14 09:28:13,573 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26692.55 MB 2025-02-14 09:28:13,573 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26692.55 MB 2025-02-14 09:28:13,573 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:28:13,573 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24708.72 MB 2025-02-14 09:28:13,630 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 09:28:13,631 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 09:28:13,631 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-14 09:28:13,631 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:28:13,631 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24168.11 MB 2025-02-14 09:28:13,631 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24327.11 MB 2025-02-14 09:28:13,631 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 159.00 MB 2025-02-14 09:28:13,631 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26692.55 MB 2025-02-14 09:28:13,631 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26791.12 MB 2025-02-14 09:28:13,631 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 98.57 MB 2025-02-14 09:28:13,631 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24443.89 MB 2025-02-14 09:28:13,640 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 09:28:13,640 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 09:28:13,640 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:28:13,640 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:28:13,640 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24427.69 MB 2025-02-14 09:28:13,640 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24586.21 MB 2025-02-14 09:28:13,640 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 158.52 MB 2025-02-14 09:28:13,640 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26791.12 MB 2025-02-14 09:28:13,640 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26791.12 MB 2025-02-14 09:28:13,640 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:28:13,640 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24586.21 MB 2025-02-14 09:28:13,642 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 09:28:13,642 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 09:28:13,642 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.19 seconds 2025-02-14 09:28:13,642 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:28:13,642 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22057.17 MB 2025-02-14 09:28:13,642 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24728.38 MB 2025-02-14 09:28:13,642 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2671.21 MB 2025-02-14 09:28:13,642 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53248.79 MB 2025-02-14 09:28:13,642 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26791.12 MB 2025-02-14 09:28:13,642 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26457.67 MB 2025-02-14 09:28:13,642 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24728.38 MB 2025-02-14 09:28:13,855 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 09:28:13,855 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 09:28:13,855 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 09:28:13,855 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:28:13,855 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24728.38 MB 2025-02-14 09:28:13,855 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26859.51 MB 2025-02-14 09:28:13,855 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2131.13 MB 2025-02-14 09:28:13,855 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26791.12 MB 2025-02-14 09:28:13,855 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27852.28 MB 2025-02-14 09:28:13,855 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1061.16 MB 2025-02-14 09:28:13,855 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27072.60 MB 2025-02-14 09:28:13,870 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 5767, cut from 5769 2025-02-14 09:28:13,870 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 09:28:13,878 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 09:28:13,878 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 09:28:13,878 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:28:13,878 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:28:13,878 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26859.51 MB 2025-02-14 09:28:13,878 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32826.20 MB 2025-02-14 09:28:13,878 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5966.68 MB 2025-02-14 09:28:13,878 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27852.28 MB 2025-02-14 09:28:13,878 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35269.90 MB 2025-02-14 09:28:13,878 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7417.63 MB 2025-02-14 09:28:13,878 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32826.20 MB 2025-02-14 09:28:14,062 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 5559] 2025-02-14 09:28:14,064 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:28:14,064 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 09:28:14,066 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:28:14,066 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 09:28:14,074 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 09:28:14,076 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:28:14,076 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 09:28:14,076 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 09:29:35,651 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:29:35,651 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 09:29:35,656 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 09:29:35,660 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:29:35,660 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 349, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 09:29:35,661 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:29:35,661 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 349, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 09:29:40,990 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 09:29:40,990 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 09:29:40,990 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.33 seconds 2025-02-14 09:29:40,990 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:29:40,990 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23464.74 MB 2025-02-14 09:29:40,990 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24699.96 MB 2025-02-14 09:29:40,990 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1235.22 MB 2025-02-14 09:29:40,990 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44168.12 MB 2025-02-14 09:29:40,990 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27927.77 MB 2025-02-14 09:29:40,990 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16240.35 MB 2025-02-14 09:29:40,990 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33615.59 MB 2025-02-14 09:29:41,012 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 09:29:41,013 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 09:29:41,013 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:29:41,013 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:29:41,013 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24699.96 MB 2025-02-14 09:29:41,013 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25208.52 MB 2025-02-14 09:29:41,013 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 508.56 MB 2025-02-14 09:29:41,013 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27927.77 MB 2025-02-14 09:29:41,013 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32054.97 MB 2025-02-14 09:29:41,013 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4127.20 MB 2025-02-14 09:29:41,013 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29467.71 MB 2025-02-14 09:29:42,606 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 09:29:42,606 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 09:29:42,606 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.59 seconds 2025-02-14 09:29:42,606 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:29:42,606 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25208.52 MB 2025-02-14 09:29:42,606 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25654.43 MB 2025-02-14 09:29:42,606 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 445.91 MB 2025-02-14 09:29:42,606 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32054.97 MB 2025-02-14 09:29:42,606 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29053.94 MB 2025-02-14 09:29:42,606 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3001.02 MB 2025-02-14 09:29:42,606 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29634.80 MB 2025-02-14 09:29:42,618 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 09:29:42,618 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 09:29:42,618 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:29:42,618 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:29:42,618 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25654.43 MB 2025-02-14 09:29:42,618 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27241.79 MB 2025-02-14 09:29:42,618 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1587.36 MB 2025-02-14 09:29:42,618 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29053.94 MB 2025-02-14 09:29:42,618 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30639.39 MB 2025-02-14 09:29:42,618 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1585.45 MB 2025-02-14 09:29:42,618 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28432.43 MB 2025-02-14 09:29:42,798 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 09:29:42,798 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 09:29:42,798 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-14 09:29:42,798 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:29:42,798 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27241.79 MB 2025-02-14 09:29:42,798 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29124.96 MB 2025-02-14 09:29:42,798 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1883.17 MB 2025-02-14 09:29:42,798 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30639.39 MB 2025-02-14 09:29:42,798 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35792.09 MB 2025-02-14 09:29:42,798 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5152.70 MB 2025-02-14 09:29:42,798 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33782.15 MB 2025-02-14 09:29:42,799 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 09:29:42,799 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 09:29:42,799 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.19 seconds 2025-02-14 09:29:42,799 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:29:42,799 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25654.43 MB 2025-02-14 09:29:42,799 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29124.96 MB 2025-02-14 09:29:42,799 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3470.53 MB 2025-02-14 09:29:42,799 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29053.94 MB 2025-02-14 09:29:42,799 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35792.09 MB 2025-02-14 09:29:42,799 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6738.15 MB 2025-02-14 09:29:42,799 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33782.15 MB 2025-02-14 09:29:42,945 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 09:29:42,945 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 09:29:42,945 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-14 09:29:42,945 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:29:42,945 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30413.14 MB 2025-02-14 09:29:42,945 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31057.42 MB 2025-02-14 09:29:42,945 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 644.28 MB 2025-02-14 09:29:42,945 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35792.09 MB 2025-02-14 09:29:42,945 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36140.22 MB 2025-02-14 09:29:42,945 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 348.13 MB 2025-02-14 09:29:42,945 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31651.96 MB 2025-02-14 09:29:42,961 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 09:29:42,961 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 09:29:42,961 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:29:42,961 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:29:42,961 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31404.25 MB 2025-02-14 09:29:42,961 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31631.69 MB 2025-02-14 09:29:42,961 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.44 MB 2025-02-14 09:29:42,961 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36140.22 MB 2025-02-14 09:29:42,961 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36140.22 MB 2025-02-14 09:29:42,961 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:29:42,961 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31789.43 MB 2025-02-14 09:29:42,962 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 09:29:42,962 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 09:29:42,962 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.30 seconds 2025-02-14 09:29:42,962 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:29:42,962 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22248.80 MB 2025-02-14 09:29:42,962 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31832.76 MB 2025-02-14 09:29:42,962 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 9583.97 MB 2025-02-14 09:29:42,962 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44168.12 MB 2025-02-14 09:29:42,962 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36140.22 MB 2025-02-14 09:29:42,962 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8027.90 MB 2025-02-14 09:29:42,962 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31832.76 MB 2025-02-14 09:29:43,228 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 09:29:43,228 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 09:29:43,228 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 09:29:43,228 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:29:43,228 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31832.76 MB 2025-02-14 09:29:43,228 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34846.80 MB 2025-02-14 09:29:43,228 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-14 09:29:43,228 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36140.22 MB 2025-02-14 09:29:43,228 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36274.44 MB 2025-02-14 09:29:43,228 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 134.22 MB 2025-02-14 09:29:43,228 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35148.42 MB 2025-02-14 09:29:43,246 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 09:29:43,246 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2,'] 2025-02-14 09:29:43,253 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 09:29:43,253 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 09:29:43,253 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:29:43,253 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:29:43,253 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26951.15 MB 2025-02-14 09:29:43,253 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35390.17 MB 2025-02-14 09:29:43,253 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 09:29:43,253 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36274.44 MB 2025-02-14 09:29:43,253 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46764.39 MB 2025-02-14 09:29:43,253 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 09:29:43,253 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35390.17 MB 2025-02-14 09:29:43,422 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 09:29:43,423 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:29:43,423 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 09:29:43,424 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:29:43,424 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 09:29:43,429 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 09:29:43,430 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:29:43,430 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 09:29:43,431 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2,'] 2025-02-14 09:31:15,373 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:31:15,373 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 09:31:15,378 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 09:31:15,382 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:31:15,382 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1869, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 09:31:15,383 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:31:15,383 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1869, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 09:31:43,973 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 09:31:43,973 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 09:31:43,973 - resource_logging.py:150 - __exit__ - DEBUG - Time: 28.58 seconds 2025-02-14 09:31:43,973 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:31:43,973 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34056.34 MB 2025-02-14 09:31:43,973 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40670.76 MB 2025-02-14 09:31:43,973 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6614.42 MB 2025-02-14 09:31:43,973 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59349.40 MB 2025-02-14 09:31:43,973 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50088.38 MB 2025-02-14 09:31:43,974 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9261.02 MB 2025-02-14 09:31:43,974 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49643.82 MB 2025-02-14 09:31:44,085 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 09:31:44,085 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 09:31:44,085 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 09:31:44,085 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:31:44,085 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40670.76 MB 2025-02-14 09:31:44,085 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33558.34 MB 2025-02-14 09:31:44,085 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7112.42 MB 2025-02-14 09:31:44,085 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50088.38 MB 2025-02-14 09:31:44,085 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 67196.94 MB 2025-02-14 09:31:44,085 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 17108.57 MB 2025-02-14 09:31:44,085 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 59370.47 MB 2025-02-14 09:31:46,001 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 09:31:46,001 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 09:31:46,001 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 09:31:46,001 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:31:46,001 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33558.34 MB 2025-02-14 09:31:46,001 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34089.19 MB 2025-02-14 09:31:46,001 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 09:31:46,001 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 67196.94 MB 2025-02-14 09:31:46,001 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40695.23 MB 2025-02-14 09:31:46,001 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26501.71 MB 2025-02-14 09:31:46,001 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38068.52 MB 2025-02-14 09:31:46,014 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 09:31:46,014 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 09:31:46,014 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:31:46,014 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:31:46,014 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34089.19 MB 2025-02-14 09:31:46,014 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35978.54 MB 2025-02-14 09:31:46,014 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.35 MB 2025-02-14 09:31:46,014 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40695.23 MB 2025-02-14 09:31:46,014 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40695.23 MB 2025-02-14 09:31:46,014 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:31:46,014 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37395.97 MB 2025-02-14 09:31:46,225 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 09:31:46,225 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 09:31:46,225 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 09:31:46,225 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:31:46,225 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35978.54 MB 2025-02-14 09:31:46,225 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38220.40 MB 2025-02-14 09:31:46,225 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 09:31:46,225 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40695.23 MB 2025-02-14 09:31:46,225 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46357.54 MB 2025-02-14 09:31:46,225 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 09:31:46,225 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43765.58 MB 2025-02-14 09:31:46,226 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 09:31:46,226 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 09:31:46,226 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 09:31:46,226 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:31:46,226 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34089.19 MB 2025-02-14 09:31:46,226 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38220.40 MB 2025-02-14 09:31:46,226 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.21 MB 2025-02-14 09:31:46,226 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40695.23 MB 2025-02-14 09:31:46,226 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46357.54 MB 2025-02-14 09:31:46,226 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 09:31:46,226 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43765.58 MB 2025-02-14 09:31:46,393 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 09:31:46,393 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 09:31:46,393 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 09:31:46,393 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:31:46,393 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39754.84 MB 2025-02-14 09:31:46,393 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40521.84 MB 2025-02-14 09:31:46,393 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 09:31:46,393 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46357.54 MB 2025-02-14 09:31:46,393 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46770.68 MB 2025-02-14 09:31:46,393 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 09:31:46,393 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41229.63 MB 2025-02-14 09:31:46,412 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 09:31:46,412 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 09:31:46,412 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:31:46,412 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:31:46,412 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40934.73 MB 2025-02-14 09:31:46,412 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41164.54 MB 2025-02-14 09:31:46,412 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.81 MB 2025-02-14 09:31:46,412 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46770.68 MB 2025-02-14 09:31:46,412 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46770.68 MB 2025-02-14 09:31:46,412 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:31:46,412 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41368.45 MB 2025-02-14 09:31:46,413 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 09:31:46,413 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 09:31:46,413 - resource_logging.py:150 - __exit__ - DEBUG - Time: 31.03 seconds 2025-02-14 09:31:46,413 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:31:46,413 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27544.60 MB 2025-02-14 09:31:46,413 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41365.56 MB 2025-02-14 09:31:46,413 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13820.96 MB 2025-02-14 09:31:46,413 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59349.40 MB 2025-02-14 09:31:46,413 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46770.68 MB 2025-02-14 09:31:46,413 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12578.72 MB 2025-02-14 09:31:46,413 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41368.45 MB 2025-02-14 09:31:46,682 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 09:31:46,682 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 09:31:46,682 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 09:31:46,682 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:31:46,682 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41365.56 MB 2025-02-14 09:31:46,682 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32549.15 MB 2025-02-14 09:31:46,682 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8816.41 MB 2025-02-14 09:31:46,682 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46770.68 MB 2025-02-14 09:31:46,682 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46770.68 MB 2025-02-14 09:31:46,682 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:31:46,682 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43876.61 MB 2025-02-14 09:31:46,700 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8160, cut from 8162 2025-02-14 09:31:46,700 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 09:31:46,706 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 09:31:46,706 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 09:31:46,706 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:31:46,706 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:31:46,706 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32549.15 MB 2025-02-14 09:31:46,706 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40986.62 MB 2025-02-14 09:31:46,706 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8437.47 MB 2025-02-14 09:31:46,706 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46770.68 MB 2025-02-14 09:31:46,706 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50964.99 MB 2025-02-14 09:31:46,706 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4194.30 MB 2025-02-14 09:31:46,706 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40986.62 MB 2025-02-14 09:31:46,869 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7952] 2025-02-14 09:31:46,870 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:31:46,870 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 09:31:46,871 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:31:46,871 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 09:31:46,876 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 09:31:46,877 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:31:46,877 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 09:31:46,877 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 09:32:58,167 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:32:58,168 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 09:32:58,173 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 09:32:58,176 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:32:58,176 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2002, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 09:32:58,177 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:32:58,177 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2002, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 09:33:28,915 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 09:33:28,915 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 09:33:28,915 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.73 seconds 2025-02-14 09:33:28,915 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:33:28,915 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34983.11 MB 2025-02-14 09:33:28,915 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42068.07 MB 2025-02-14 09:33:28,915 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7084.97 MB 2025-02-14 09:33:28,915 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59353.60 MB 2025-02-14 09:33:28,915 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50556.04 MB 2025-02-14 09:33:28,915 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8797.55 MB 2025-02-14 09:33:28,915 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51023.57 MB 2025-02-14 09:33:29,049 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 09:33:29,049 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 09:33:29,049 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 09:33:29,049 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:33:29,049 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42068.07 MB 2025-02-14 09:33:29,049 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34249.77 MB 2025-02-14 09:33:29,049 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7818.30 MB 2025-02-14 09:33:29,049 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50556.04 MB 2025-02-14 09:33:29,049 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 70634.18 MB 2025-02-14 09:33:29,049 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 20078.13 MB 2025-02-14 09:33:29,049 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 62230.28 MB 2025-02-14 09:33:30,973 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 09:33:30,973 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 09:33:30,973 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 09:33:30,973 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:33:30,973 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34249.77 MB 2025-02-14 09:33:30,973 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34780.61 MB 2025-02-14 09:33:30,974 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 09:33:30,974 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 70634.18 MB 2025-02-14 09:33:30,974 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40691.04 MB 2025-02-14 09:33:30,974 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29943.14 MB 2025-02-14 09:33:30,974 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38759.95 MB 2025-02-14 09:33:30,990 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 09:33:30,990 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 09:33:30,990 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:33:30,990 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:33:30,990 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34780.61 MB 2025-02-14 09:33:30,990 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36669.97 MB 2025-02-14 09:33:30,990 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.35 MB 2025-02-14 09:33:30,990 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40691.04 MB 2025-02-14 09:33:30,990 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42578.48 MB 2025-02-14 09:33:30,990 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 09:33:30,990 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38087.39 MB 2025-02-14 09:33:31,200 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 09:33:31,200 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 09:33:31,200 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 09:33:31,200 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:33:31,200 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36669.97 MB 2025-02-14 09:33:31,200 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38911.82 MB 2025-02-14 09:33:31,200 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 09:33:31,200 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42578.48 MB 2025-02-14 09:33:31,200 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48240.79 MB 2025-02-14 09:33:31,200 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 09:33:31,200 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44457.00 MB 2025-02-14 09:33:31,201 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 09:33:31,201 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 09:33:31,201 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 09:33:31,201 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:33:31,201 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34780.61 MB 2025-02-14 09:33:31,201 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38911.82 MB 2025-02-14 09:33:31,201 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.21 MB 2025-02-14 09:33:31,201 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40691.04 MB 2025-02-14 09:33:31,201 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48240.79 MB 2025-02-14 09:33:31,201 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-14 09:33:31,201 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44457.00 MB 2025-02-14 09:33:31,367 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 09:33:31,367 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 09:33:31,367 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 09:33:31,367 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:33:31,367 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40446.26 MB 2025-02-14 09:33:31,367 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41213.27 MB 2025-02-14 09:33:31,367 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 09:33:31,367 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48240.79 MB 2025-02-14 09:33:31,367 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48656.02 MB 2025-02-14 09:33:31,367 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 09:33:31,367 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41921.05 MB 2025-02-14 09:33:31,386 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 09:33:31,386 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 09:33:31,386 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:33:31,386 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:33:31,386 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41626.15 MB 2025-02-14 09:33:31,386 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41854.62 MB 2025-02-14 09:33:31,386 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.47 MB 2025-02-14 09:33:31,386 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48656.02 MB 2025-02-14 09:33:31,386 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48656.02 MB 2025-02-14 09:33:31,386 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:33:31,386 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42055.14 MB 2025-02-14 09:33:31,387 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 09:33:31,387 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 09:33:31,387 - resource_logging.py:150 - __exit__ - DEBUG - Time: 33.21 seconds 2025-02-14 09:33:31,387 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:33:31,387 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28007.98 MB 2025-02-14 09:33:31,387 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42055.01 MB 2025-02-14 09:33:31,387 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14047.03 MB 2025-02-14 09:33:31,387 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59353.60 MB 2025-02-14 09:33:31,387 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48656.02 MB 2025-02-14 09:33:31,387 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10697.57 MB 2025-02-14 09:33:31,387 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42055.14 MB 2025-02-14 09:33:31,655 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 09:33:31,655 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 09:33:31,655 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 09:33:31,655 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:33:31,655 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42055.01 MB 2025-02-14 09:33:31,655 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33002.95 MB 2025-02-14 09:33:31,655 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9052.06 MB 2025-02-14 09:33:31,655 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48656.02 MB 2025-02-14 09:33:31,655 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48656.02 MB 2025-02-14 09:33:31,655 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:33:31,655 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44558.07 MB 2025-02-14 09:33:31,673 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8134, cut from 8136 2025-02-14 09:33:31,673 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 09:33:31,679 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 09:33:31,679 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 09:33:31,679 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:33:31,679 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:33:31,679 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33002.95 MB 2025-02-14 09:33:31,679 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41412.75 MB 2025-02-14 09:33:31,679 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8409.81 MB 2025-02-14 09:33:31,679 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48656.02 MB 2025-02-14 09:33:31,679 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57017.37 MB 2025-02-14 09:33:31,679 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8361.35 MB 2025-02-14 09:33:31,679 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41412.75 MB 2025-02-14 09:33:31,840 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7926] 2025-02-14 09:33:31,842 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:33:31,842 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 09:33:31,843 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:33:31,843 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 09:33:31,848 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 09:33:31,849 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:33:31,849 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 09:33:31,849 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 09:34:40,444 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:34:40,444 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 09:34:40,449 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 09:34:40,453 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:34:40,453 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1537, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 09:34:40,454 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:34:40,454 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1537, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 09:35:04,151 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 09:35:04,151 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 09:35:04,151 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.69 seconds 2025-02-14 09:35:04,151 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:35:04,151 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31742.91 MB 2025-02-14 09:35:04,151 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37182.93 MB 2025-02-14 09:35:04,151 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5440.01 MB 2025-02-14 09:35:04,151 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 69558.34 MB 2025-02-14 09:35:04,151 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48865.74 MB 2025-02-14 09:35:04,151 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20692.60 MB 2025-02-14 09:35:04,151 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46197.92 MB 2025-02-14 09:35:04,237 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 09:35:04,237 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 09:35:04,237 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 09:35:04,237 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:35:04,237 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37182.93 MB 2025-02-14 09:35:04,237 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31832.38 MB 2025-02-14 09:35:04,237 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5350.55 MB 2025-02-14 09:35:04,237 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48865.74 MB 2025-02-14 09:35:04,237 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 59303.26 MB 2025-02-14 09:35:04,237 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10437.53 MB 2025-02-14 09:35:04,237 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52882.91 MB 2025-02-14 09:35:06,155 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 09:35:06,155 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 09:35:06,155 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 09:35:06,155 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:35:06,155 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31832.38 MB 2025-02-14 09:35:06,155 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32363.22 MB 2025-02-14 09:35:06,155 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 09:35:06,155 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59303.26 MB 2025-02-14 09:35:06,155 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39246.10 MB 2025-02-14 09:35:06,155 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20057.16 MB 2025-02-14 09:35:06,155 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36342.55 MB 2025-02-14 09:35:06,171 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 09:35:06,171 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 09:35:06,171 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:35:06,171 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:35:06,171 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32363.22 MB 2025-02-14 09:35:06,171 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34252.57 MB 2025-02-14 09:35:06,171 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.35 MB 2025-02-14 09:35:06,171 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39246.10 MB 2025-02-14 09:35:06,171 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40189.82 MB 2025-02-14 09:35:06,171 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 09:35:06,171 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35670.00 MB 2025-02-14 09:35:06,381 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 09:35:06,381 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 09:35:06,381 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 09:35:06,381 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:35:06,381 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34252.57 MB 2025-02-14 09:35:06,381 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36494.43 MB 2025-02-14 09:35:06,381 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 09:35:06,381 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40189.82 MB 2025-02-14 09:35:06,381 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45852.13 MB 2025-02-14 09:35:06,381 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 09:35:06,381 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42039.61 MB 2025-02-14 09:35:06,382 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 09:35:06,382 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 09:35:06,382 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 09:35:06,382 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:35:06,382 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32363.22 MB 2025-02-14 09:35:06,382 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36494.43 MB 2025-02-14 09:35:06,382 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.21 MB 2025-02-14 09:35:06,382 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39246.10 MB 2025-02-14 09:35:06,382 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45852.13 MB 2025-02-14 09:35:06,382 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 09:35:06,382 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42039.61 MB 2025-02-14 09:35:06,550 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 09:35:06,550 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 09:35:06,550 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 09:35:06,550 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:35:06,550 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38028.87 MB 2025-02-14 09:35:06,550 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38795.87 MB 2025-02-14 09:35:06,550 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 09:35:06,550 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45852.13 MB 2025-02-14 09:35:06,550 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46267.37 MB 2025-02-14 09:35:06,550 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 09:35:06,550 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39503.66 MB 2025-02-14 09:35:06,569 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 09:35:06,569 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 09:35:06,569 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:35:06,569 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:35:06,569 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39208.76 MB 2025-02-14 09:35:06,569 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39437.77 MB 2025-02-14 09:35:06,569 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.01 MB 2025-02-14 09:35:06,569 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46267.37 MB 2025-02-14 09:35:06,569 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46267.37 MB 2025-02-14 09:35:06,569 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:35:06,569 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39655.34 MB 2025-02-14 09:35:06,570 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 09:35:06,570 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 09:35:06,570 - resource_logging.py:150 - __exit__ - DEBUG - Time: 26.11 seconds 2025-02-14 09:35:06,570 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:35:06,570 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26387.88 MB 2025-02-14 09:35:06,570 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39638.70 MB 2025-02-14 09:35:06,570 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13250.82 MB 2025-02-14 09:35:06,570 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 69558.34 MB 2025-02-14 09:35:06,570 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46267.37 MB 2025-02-14 09:35:06,570 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23290.97 MB 2025-02-14 09:35:06,570 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39655.34 MB 2025-02-14 09:35:06,839 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 09:35:06,839 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 09:35:06,839 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 09:35:06,839 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:35:06,839 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39638.70 MB 2025-02-14 09:35:06,839 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31390.96 MB 2025-02-14 09:35:06,839 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8247.74 MB 2025-02-14 09:35:06,840 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46267.37 MB 2025-02-14 09:35:06,840 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46267.37 MB 2025-02-14 09:35:06,840 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:35:06,840 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42148.52 MB 2025-02-14 09:35:06,857 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8156, cut from 8158 2025-02-14 09:35:06,858 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 09:35:06,864 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 09:35:06,864 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 09:35:06,864 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:35:06,864 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:35:06,864 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31390.96 MB 2025-02-14 09:35:06,864 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39824.26 MB 2025-02-14 09:35:06,864 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8433.30 MB 2025-02-14 09:35:06,864 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46267.37 MB 2025-02-14 09:35:06,864 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54651.78 MB 2025-02-14 09:35:06,864 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8384.41 MB 2025-02-14 09:35:06,864 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39824.26 MB 2025-02-14 09:35:07,032 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7948] 2025-02-14 09:35:07,034 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:35:07,034 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 09:35:07,034 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:35:07,035 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 09:35:07,039 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 09:35:07,041 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:35:07,041 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 09:35:07,041 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 09:35:58,530 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:35:58,530 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 09:35:58,535 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 09:35:58,539 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:35:58,539 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1677, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 09:35:58,540 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:35:58,540 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1677, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 09:36:24,478 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 09:36:24,478 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 09:36:24,478 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.93 seconds 2025-02-14 09:36:24,478 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:36:24,478 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32718.46 MB 2025-02-14 09:36:24,478 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38653.40 MB 2025-02-14 09:36:24,478 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5934.94 MB 2025-02-14 09:36:24,478 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63036.19 MB 2025-02-14 09:36:24,478 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49371.15 MB 2025-02-14 09:36:24,478 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13665.04 MB 2025-02-14 09:36:24,478 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47626.45 MB 2025-02-14 09:36:24,571 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 09:36:24,571 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 09:36:24,571 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 09:36:24,571 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:36:24,571 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38653.40 MB 2025-02-14 09:36:24,571 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32560.20 MB 2025-02-14 09:36:24,571 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6093.20 MB 2025-02-14 09:36:24,571 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49371.15 MB 2025-02-14 09:36:24,571 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 63034.10 MB 2025-02-14 09:36:24,571 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 13662.95 MB 2025-02-14 09:36:24,571 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 55749.30 MB 2025-02-14 09:36:26,495 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 09:36:26,495 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 09:36:26,495 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 09:36:26,495 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:36:26,495 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32560.20 MB 2025-02-14 09:36:26,495 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33091.04 MB 2025-02-14 09:36:26,495 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 09:36:26,495 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63034.10 MB 2025-02-14 09:36:26,495 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44851.79 MB 2025-02-14 09:36:26,495 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18182.31 MB 2025-02-14 09:36:26,495 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37070.37 MB 2025-02-14 09:36:26,508 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 09:36:26,508 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 09:36:26,508 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:36:26,508 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:36:26,508 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33091.04 MB 2025-02-14 09:36:26,508 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34980.39 MB 2025-02-14 09:36:26,508 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.35 MB 2025-02-14 09:36:26,508 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44851.79 MB 2025-02-14 09:36:26,508 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44851.79 MB 2025-02-14 09:36:26,508 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:36:26,508 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36397.82 MB 2025-02-14 09:36:26,720 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 09:36:26,720 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 09:36:26,720 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 09:36:26,720 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:36:26,720 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34980.39 MB 2025-02-14 09:36:26,720 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37222.25 MB 2025-02-14 09:36:26,720 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 09:36:26,720 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44851.79 MB 2025-02-14 09:36:26,720 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46739.23 MB 2025-02-14 09:36:26,720 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 09:36:26,720 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42767.43 MB 2025-02-14 09:36:26,721 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 09:36:26,721 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 09:36:26,721 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 09:36:26,721 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:36:26,721 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33091.04 MB 2025-02-14 09:36:26,721 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37222.25 MB 2025-02-14 09:36:26,721 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.21 MB 2025-02-14 09:36:26,721 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44851.79 MB 2025-02-14 09:36:26,721 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46739.23 MB 2025-02-14 09:36:26,721 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 09:36:26,721 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42767.43 MB 2025-02-14 09:36:26,889 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 09:36:26,889 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 09:36:26,889 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 09:36:26,889 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:36:26,889 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38756.69 MB 2025-02-14 09:36:26,889 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39523.69 MB 2025-02-14 09:36:26,889 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 09:36:26,889 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46739.23 MB 2025-02-14 09:36:26,889 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47154.46 MB 2025-02-14 09:36:26,889 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 09:36:26,889 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40231.48 MB 2025-02-14 09:36:26,908 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 09:36:26,908 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 09:36:26,908 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:36:26,908 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:36:26,908 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39936.58 MB 2025-02-14 09:36:26,908 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40165.47 MB 2025-02-14 09:36:26,908 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.89 MB 2025-02-14 09:36:26,908 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47154.46 MB 2025-02-14 09:36:26,908 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47154.46 MB 2025-02-14 09:36:26,908 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:36:26,908 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40406.73 MB 2025-02-14 09:36:26,909 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 09:36:26,909 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 09:36:26,909 - resource_logging.py:150 - __exit__ - DEBUG - Time: 28.37 seconds 2025-02-14 09:36:26,909 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:36:26,909 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26875.65 MB 2025-02-14 09:36:26,909 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40366.27 MB 2025-02-14 09:36:26,909 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13490.62 MB 2025-02-14 09:36:26,909 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63036.19 MB 2025-02-14 09:36:26,909 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47154.46 MB 2025-02-14 09:36:26,909 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15881.73 MB 2025-02-14 09:36:26,909 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40406.73 MB 2025-02-14 09:36:27,180 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 09:36:27,180 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 09:36:27,180 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 09:36:27,180 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:36:27,180 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40366.27 MB 2025-02-14 09:36:27,180 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31876.89 MB 2025-02-14 09:36:27,180 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8489.38 MB 2025-02-14 09:36:27,180 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47154.46 MB 2025-02-14 09:36:27,180 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47154.46 MB 2025-02-14 09:36:27,181 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:36:27,181 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42874.56 MB 2025-02-14 09:36:27,198 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8151, cut from 8153 2025-02-14 09:36:27,198 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 09:36:27,204 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 09:36:27,204 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 09:36:27,204 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:36:27,204 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:36:27,204 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31876.89 MB 2025-02-14 09:36:27,204 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40304.22 MB 2025-02-14 09:36:27,204 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8427.34 MB 2025-02-14 09:36:27,204 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47154.46 MB 2025-02-14 09:36:27,204 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55534.68 MB 2025-02-14 09:36:27,205 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8380.22 MB 2025-02-14 09:36:27,205 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40304.22 MB 2025-02-14 09:36:27,368 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7943] 2025-02-14 09:36:27,370 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:36:27,370 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 09:36:27,371 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:36:27,371 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 09:36:27,376 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 09:36:27,377 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:36:27,377 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 09:36:27,377 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 09:36:58,429 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:36:58,429 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 09:36:58,434 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 09:36:58,438 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:36:58,438 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1023, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 09:36:58,439 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:36:58,439 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1023, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 09:37:14,307 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 09:37:14,307 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 09:37:14,307 - resource_logging.py:150 - __exit__ - DEBUG - Time: 15.86 seconds 2025-02-14 09:37:14,307 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:37:14,307 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28161.28 MB 2025-02-14 09:37:14,307 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31781.62 MB 2025-02-14 09:37:14,307 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3620.34 MB 2025-02-14 09:37:14,307 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63914.90 MB 2025-02-14 09:37:14,307 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38675.68 MB 2025-02-14 09:37:14,307 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25239.22 MB 2025-02-14 09:37:14,307 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40659.78 MB 2025-02-14 09:37:14,365 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 09:37:14,365 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 09:37:14,365 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 09:37:14,365 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:37:14,365 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31781.62 MB 2025-02-14 09:37:14,365 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29161.30 MB 2025-02-14 09:37:14,365 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2620.32 MB 2025-02-14 09:37:14,365 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38675.68 MB 2025-02-14 09:37:14,365 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46948.94 MB 2025-02-14 09:37:14,365 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8273.26 MB 2025-02-14 09:37:14,365 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42206.60 MB 2025-02-14 09:37:16,285 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 09:37:16,285 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 09:37:16,285 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 09:37:16,285 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:37:16,286 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29161.30 MB 2025-02-14 09:37:16,286 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29692.14 MB 2025-02-14 09:37:16,286 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 09:37:16,286 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46948.94 MB 2025-02-14 09:37:16,286 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37178.31 MB 2025-02-14 09:37:16,286 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9770.63 MB 2025-02-14 09:37:16,286 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33671.48 MB 2025-02-14 09:37:16,300 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 09:37:16,300 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 09:37:16,300 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:37:16,300 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:37:16,300 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29692.14 MB 2025-02-14 09:37:16,300 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31581.50 MB 2025-02-14 09:37:16,300 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.35 MB 2025-02-14 09:37:16,300 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37178.31 MB 2025-02-14 09:37:16,300 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38122.03 MB 2025-02-14 09:37:16,300 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 09:37:16,300 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32998.93 MB 2025-02-14 09:37:16,511 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 09:37:16,511 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 09:37:16,511 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 09:37:16,511 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:37:16,511 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31581.50 MB 2025-02-14 09:37:16,511 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33823.35 MB 2025-02-14 09:37:16,511 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 09:37:16,511 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38122.03 MB 2025-02-14 09:37:16,511 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43784.34 MB 2025-02-14 09:37:16,511 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 09:37:16,511 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39368.54 MB 2025-02-14 09:37:16,512 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 09:37:16,512 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 09:37:16,512 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 09:37:16,512 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:37:16,512 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29692.14 MB 2025-02-14 09:37:16,512 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33823.35 MB 2025-02-14 09:37:16,512 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.21 MB 2025-02-14 09:37:16,512 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37178.31 MB 2025-02-14 09:37:16,512 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43784.34 MB 2025-02-14 09:37:16,512 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 09:37:16,512 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39368.54 MB 2025-02-14 09:37:16,683 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 09:37:16,683 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 09:37:16,683 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 09:37:16,683 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:37:16,683 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35357.80 MB 2025-02-14 09:37:16,683 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36124.80 MB 2025-02-14 09:37:16,683 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 09:37:16,683 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43784.34 MB 2025-02-14 09:37:16,683 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44197.48 MB 2025-02-14 09:37:16,683 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 09:37:16,683 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36832.59 MB 2025-02-14 09:37:16,702 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 09:37:16,702 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 09:37:16,702 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:37:16,702 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:37:16,702 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36537.69 MB 2025-02-14 09:37:16,702 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36766.72 MB 2025-02-14 09:37:16,702 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.03 MB 2025-02-14 09:37:16,702 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44197.48 MB 2025-02-14 09:37:16,702 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44197.48 MB 2025-02-14 09:37:16,702 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:37:16,702 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36960.14 MB 2025-02-14 09:37:16,703 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 09:37:16,703 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 09:37:16,703 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.26 seconds 2025-02-14 09:37:16,703 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:37:16,703 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24597.07 MB 2025-02-14 09:37:16,703 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36967.67 MB 2025-02-14 09:37:16,703 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12370.61 MB 2025-02-14 09:37:16,703 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63914.90 MB 2025-02-14 09:37:16,703 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44197.48 MB 2025-02-14 09:37:16,703 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19717.42 MB 2025-02-14 09:37:16,703 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36967.67 MB 2025-02-14 09:37:16,972 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 09:37:16,972 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 09:37:16,972 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 09:37:16,972 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:37:16,972 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36967.67 MB 2025-02-14 09:37:16,972 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29600.76 MB 2025-02-14 09:37:16,972 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7366.91 MB 2025-02-14 09:37:16,972 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44197.48 MB 2025-02-14 09:37:16,972 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44197.48 MB 2025-02-14 09:37:16,972 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:37:16,972 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39477.80 MB 2025-02-14 09:37:16,990 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8157, cut from 8159 2025-02-14 09:37:16,990 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 09:37:16,996 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 09:37:16,996 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 09:37:16,996 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:37:16,996 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:37:16,996 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29600.76 MB 2025-02-14 09:37:16,997 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38035.38 MB 2025-02-14 09:37:16,997 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8434.62 MB 2025-02-14 09:37:16,997 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44197.48 MB 2025-02-14 09:37:16,997 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52581.89 MB 2025-02-14 09:37:16,997 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8384.41 MB 2025-02-14 09:37:16,997 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38035.38 MB 2025-02-14 09:37:17,161 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7949] 2025-02-14 09:37:17,162 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:37:17,162 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 09:37:17,163 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:37:17,163 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 09:37:17,168 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 09:37:17,169 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:37:17,169 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 09:37:17,169 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 09:37:27,396 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:37:27,396 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 09:37:27,401 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 09:37:27,405 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:37:27,405 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 806, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 09:37:27,406 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:37:27,406 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 806, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 09:37:40,018 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 09:37:40,019 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 09:37:40,019 - resource_logging.py:150 - __exit__ - DEBUG - Time: 12.61 seconds 2025-02-14 09:37:40,019 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:37:40,019 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26649.19 MB 2025-02-14 09:37:40,019 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29501.58 MB 2025-02-14 09:37:40,019 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2852.39 MB 2025-02-14 09:37:40,019 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60966.31 MB 2025-02-14 09:37:40,019 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35144.07 MB 2025-02-14 09:37:40,019 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25822.23 MB 2025-02-14 09:37:40,019 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38386.29 MB 2025-02-14 09:37:40,046 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 09:37:40,047 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 09:37:40,047 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 09:37:40,047 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:37:40,047 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29501.58 MB 2025-02-14 09:37:40,047 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25651.33 MB 2025-02-14 09:37:40,047 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3850.24 MB 2025-02-14 09:37:40,047 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35144.07 MB 2025-02-14 09:37:40,047 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35144.07 MB 2025-02-14 09:37:40,047 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:37:40,047 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30379.62 MB 2025-02-14 09:37:40,344 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 09:37:40,344 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 09:37:40,344 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.30 seconds 2025-02-14 09:37:40,344 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:37:40,344 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25651.33 MB 2025-02-14 09:37:40,344 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25732.29 MB 2025-02-14 09:37:40,344 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 80.95 MB 2025-02-14 09:37:40,344 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35144.07 MB 2025-02-14 09:37:40,344 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32289.85 MB 2025-02-14 09:37:40,344 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2854.22 MB 2025-02-14 09:37:40,344 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29544.59 MB 2025-02-14 09:37:40,348 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 09:37:40,348 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 09:37:40,348 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 09:37:40,348 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:37:40,348 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25732.29 MB 2025-02-14 09:37:40,348 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26020.37 MB 2025-02-14 09:37:40,348 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 288.08 MB 2025-02-14 09:37:40,349 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32289.85 MB 2025-02-14 09:37:40,349 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32289.85 MB 2025-02-14 09:37:40,349 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:37:40,349 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26236.54 MB 2025-02-14 09:37:40,409 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 09:37:40,409 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 09:37:40,409 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 09:37:40,409 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:37:40,409 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26020.37 MB 2025-02-14 09:37:40,409 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26370.30 MB 2025-02-14 09:37:40,409 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 349.93 MB 2025-02-14 09:37:40,409 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32289.85 MB 2025-02-14 09:37:40,409 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32289.85 MB 2025-02-14 09:37:40,409 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:37:40,409 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27207.76 MB 2025-02-14 09:37:40,410 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 09:37:40,410 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 09:37:40,410 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 09:37:40,410 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:37:40,410 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25732.29 MB 2025-02-14 09:37:40,410 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26370.30 MB 2025-02-14 09:37:40,410 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 638.02 MB 2025-02-14 09:37:40,410 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32289.85 MB 2025-02-14 09:37:40,410 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32289.85 MB 2025-02-14 09:37:40,410 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:37:40,410 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27207.76 MB 2025-02-14 09:37:40,442 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 09:37:40,442 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 09:37:40,442 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 09:37:40,442 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:37:40,442 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26708.11 MB 2025-02-14 09:37:40,442 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26855.06 MB 2025-02-14 09:37:40,442 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 146.95 MB 2025-02-14 09:37:40,442 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32289.85 MB 2025-02-14 09:37:40,442 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32380.03 MB 2025-02-14 09:37:40,442 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 90.18 MB 2025-02-14 09:37:40,442 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26963.00 MB 2025-02-14 09:37:40,446 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 09:37:40,446 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 09:37:40,446 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 09:37:40,446 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:37:40,446 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26948.02 MB 2025-02-14 09:37:40,446 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27094.94 MB 2025-02-14 09:37:40,446 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 146.92 MB 2025-02-14 09:37:40,446 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32380.03 MB 2025-02-14 09:37:40,446 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32380.03 MB 2025-02-14 09:37:40,446 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:37:40,446 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27094.94 MB 2025-02-14 09:37:40,448 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 09:37:40,448 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 09:37:40,448 - resource_logging.py:150 - __exit__ - DEBUG - Time: 13.04 seconds 2025-02-14 09:37:40,448 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:37:40,448 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23841.02 MB 2025-02-14 09:37:40,449 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27226.78 MB 2025-02-14 09:37:40,449 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3385.76 MB 2025-02-14 09:37:40,449 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60966.31 MB 2025-02-14 09:37:40,449 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32380.03 MB 2025-02-14 09:37:40,449 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28586.28 MB 2025-02-14 09:37:40,449 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27226.78 MB 2025-02-14 09:37:40,617 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 09:37:40,617 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 09:37:40,618 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 09:37:40,618 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:37:40,618 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24196.71 MB 2025-02-14 09:37:40,618 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26173.01 MB 2025-02-14 09:37:40,618 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1976.30 MB 2025-02-14 09:37:40,618 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32380.03 MB 2025-02-14 09:37:40,618 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32380.03 MB 2025-02-14 09:37:40,618 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:37:40,618 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26370.61 MB 2025-02-14 09:37:40,630 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 5347, cut from 5349 2025-02-14 09:37:40,630 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 09:37:40,634 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 09:37:40,634 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 09:37:40,634 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:37:40,634 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:37:40,634 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26173.01 MB 2025-02-14 09:37:40,634 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31706.08 MB 2025-02-14 09:37:40,634 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5533.07 MB 2025-02-14 09:37:40,634 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32380.03 MB 2025-02-14 09:37:40,634 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35131.49 MB 2025-02-14 09:37:40,634 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2751.46 MB 2025-02-14 09:37:40,634 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31706.08 MB 2025-02-14 09:37:40,741 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 5139] 2025-02-14 09:37:40,742 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:37:40,742 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 09:37:40,743 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:37:40,743 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 09:37:40,748 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 09:37:40,749 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:37:40,749 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 09:37:40,749 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 09:38:28,544 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:38:28,544 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 09:38:28,549 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 09:38:28,552 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:38:28,552 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 182, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 09:38:28,553 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:38:28,553 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 182, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 09:38:31,355 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 09:38:31,355 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 09:38:31,355 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.80 seconds 2025-02-14 09:38:31,355 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:38:31,355 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22301.06 MB 2025-02-14 09:38:31,355 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22945.14 MB 2025-02-14 09:38:31,355 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 644.09 MB 2025-02-14 09:38:31,355 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40634.42 MB 2025-02-14 09:38:31,355 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26694.65 MB 2025-02-14 09:38:31,355 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13939.77 MB 2025-02-14 09:38:31,355 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31773.23 MB 2025-02-14 09:38:31,365 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 09:38:31,365 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 09:38:31,365 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:38:31,365 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:38:31,365 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22945.14 MB 2025-02-14 09:38:31,365 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22414.44 MB 2025-02-14 09:38:31,365 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -530.70 MB 2025-02-14 09:38:31,365 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26694.65 MB 2025-02-14 09:38:31,365 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26694.65 MB 2025-02-14 09:38:31,365 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:38:31,365 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23837.30 MB 2025-02-14 09:38:31,666 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 09:38:31,666 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 09:38:31,666 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.30 seconds 2025-02-14 09:38:31,666 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:38:31,666 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22414.44 MB 2025-02-14 09:38:31,666 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22496.72 MB 2025-02-14 09:38:31,666 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 82.28 MB 2025-02-14 09:38:31,666 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26694.65 MB 2025-02-14 09:38:31,666 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27084.72 MB 2025-02-14 09:38:31,666 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 390.07 MB 2025-02-14 09:38:31,666 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26372.52 MB 2025-02-14 09:38:31,672 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 09:38:31,672 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 09:38:31,672 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 09:38:31,672 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:38:31,672 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22496.66 MB 2025-02-14 09:38:31,672 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22789.46 MB 2025-02-14 09:38:31,672 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 292.81 MB 2025-02-14 09:38:31,672 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27084.72 MB 2025-02-14 09:38:31,672 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27084.72 MB 2025-02-14 09:38:31,672 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:38:31,672 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23009.17 MB 2025-02-14 09:38:31,734 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 09:38:31,734 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 09:38:31,734 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 09:38:31,734 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:38:31,734 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22789.46 MB 2025-02-14 09:38:31,734 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23145.15 MB 2025-02-14 09:38:31,734 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 355.68 MB 2025-02-14 09:38:31,734 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27084.72 MB 2025-02-14 09:38:31,734 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27084.72 MB 2025-02-14 09:38:31,734 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:38:31,734 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23996.33 MB 2025-02-14 09:38:31,735 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 09:38:31,735 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 09:38:31,735 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 09:38:31,735 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:38:31,735 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22496.66 MB 2025-02-14 09:38:31,735 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23145.15 MB 2025-02-14 09:38:31,735 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 648.49 MB 2025-02-14 09:38:31,735 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27084.72 MB 2025-02-14 09:38:31,735 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27084.72 MB 2025-02-14 09:38:31,735 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:38:31,735 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23996.33 MB 2025-02-14 09:38:31,768 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 09:38:31,768 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 09:38:31,768 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 09:38:31,768 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:38:31,768 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23488.62 MB 2025-02-14 09:38:31,768 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23637.98 MB 2025-02-14 09:38:31,768 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 149.36 MB 2025-02-14 09:38:31,768 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27084.72 MB 2025-02-14 09:38:31,768 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27174.90 MB 2025-02-14 09:38:31,768 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 90.18 MB 2025-02-14 09:38:31,768 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23747.69 MB 2025-02-14 09:38:31,773 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 09:38:31,773 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 09:38:31,773 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 09:38:31,773 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:38:31,773 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23732.46 MB 2025-02-14 09:38:31,773 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23881.70 MB 2025-02-14 09:38:31,773 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 149.23 MB 2025-02-14 09:38:31,773 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27174.90 MB 2025-02-14 09:38:31,773 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27174.90 MB 2025-02-14 09:38:31,773 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:38:31,773 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23881.70 MB 2025-02-14 09:38:31,774 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 09:38:31,774 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 09:38:31,774 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.22 seconds 2025-02-14 09:38:31,774 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:38:31,774 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21666.95 MB 2025-02-14 09:38:31,774 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24015.60 MB 2025-02-14 09:38:31,774 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2348.65 MB 2025-02-14 09:38:31,774 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40634.42 MB 2025-02-14 09:38:31,774 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27174.90 MB 2025-02-14 09:38:31,774 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13459.52 MB 2025-02-14 09:38:31,774 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24015.60 MB 2025-02-14 09:38:31,946 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 09:38:31,946 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 09:38:31,946 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 09:38:31,946 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:38:31,946 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24015.60 MB 2025-02-14 09:38:31,946 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24036.08 MB 2025-02-14 09:38:31,946 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 20.48 MB 2025-02-14 09:38:31,946 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27174.90 MB 2025-02-14 09:38:31,946 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27174.90 MB 2025-02-14 09:38:31,946 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:38:31,946 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25956.36 MB 2025-02-14 09:38:31,958 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 5431, cut from 5433 2025-02-14 09:38:31,959 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 09:38:31,963 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 09:38:31,963 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 09:38:31,963 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:38:31,963 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:38:31,963 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24036.08 MB 2025-02-14 09:38:31,963 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29656.36 MB 2025-02-14 09:38:31,963 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5620.28 MB 2025-02-14 09:38:31,963 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27174.90 MB 2025-02-14 09:38:31,963 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34162.61 MB 2025-02-14 09:38:31,963 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6987.71 MB 2025-02-14 09:38:31,963 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29656.36 MB 2025-02-14 09:38:32,071 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 5223] 2025-02-14 09:38:32,072 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:38:32,072 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 09:38:32,073 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:38:32,073 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 09:38:32,078 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 09:38:32,079 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:38:32,079 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 09:38:32,079 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 09:39:12,912 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:39:12,912 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 09:39:12,917 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 09:39:12,920 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:39:12,921 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1239, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 09:39:12,921 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:39:12,922 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1239, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 09:39:31,907 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 09:39:31,907 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 09:39:31,907 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.98 seconds 2025-02-14 09:39:31,907 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:39:31,907 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29666.40 MB 2025-02-14 09:39:31,907 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34051.55 MB 2025-02-14 09:39:31,907 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4385.14 MB 2025-02-14 09:39:31,907 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42544.92 MB 2025-02-14 09:39:31,907 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42255.52 MB 2025-02-14 09:39:31,907 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -289.41 MB 2025-02-14 09:39:31,907 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42988.95 MB 2025-02-14 09:39:31,954 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 09:39:31,954 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 09:39:31,954 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-14 09:39:31,954 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:39:31,954 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34051.55 MB 2025-02-14 09:39:31,954 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29917.97 MB 2025-02-14 09:39:31,954 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4133.57 MB 2025-02-14 09:39:31,954 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42255.52 MB 2025-02-14 09:39:31,954 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43488.64 MB 2025-02-14 09:39:31,954 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1233.13 MB 2025-02-14 09:39:31,954 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38964.34 MB 2025-02-14 09:39:33,612 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 09:39:33,612 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 09:39:33,612 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.66 seconds 2025-02-14 09:39:33,612 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:39:33,612 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29917.97 MB 2025-02-14 09:39:33,612 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30379.81 MB 2025-02-14 09:39:33,612 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 461.83 MB 2025-02-14 09:39:33,612 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43488.64 MB 2025-02-14 09:39:33,612 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36310.09 MB 2025-02-14 09:39:33,612 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7178.55 MB 2025-02-14 09:39:33,612 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34343.21 MB 2025-02-14 09:39:33,625 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 09:39:33,625 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 09:39:33,625 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:39:33,625 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:39:33,625 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30379.81 MB 2025-02-14 09:39:33,625 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32025.55 MB 2025-02-14 09:39:33,625 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1645.74 MB 2025-02-14 09:39:33,625 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36310.09 MB 2025-02-14 09:39:33,625 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37954.26 MB 2025-02-14 09:39:33,625 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1644.17 MB 2025-02-14 09:39:33,625 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33258.71 MB 2025-02-14 09:39:33,807 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 09:39:33,807 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 09:39:33,807 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-14 09:39:33,807 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:39:33,807 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32025.55 MB 2025-02-14 09:39:33,807 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33977.02 MB 2025-02-14 09:39:33,807 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1951.47 MB 2025-02-14 09:39:33,807 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37954.26 MB 2025-02-14 09:39:33,807 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42886.76 MB 2025-02-14 09:39:33,807 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4932.50 MB 2025-02-14 09:39:33,807 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38804.73 MB 2025-02-14 09:39:33,808 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 09:39:33,808 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 09:39:33,808 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.19 seconds 2025-02-14 09:39:33,808 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:39:33,808 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30379.81 MB 2025-02-14 09:39:33,808 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33977.02 MB 2025-02-14 09:39:33,808 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3597.21 MB 2025-02-14 09:39:33,808 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36310.09 MB 2025-02-14 09:39:33,808 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42886.76 MB 2025-02-14 09:39:33,808 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6576.67 MB 2025-02-14 09:39:33,808 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38804.73 MB 2025-02-14 09:39:33,954 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 09:39:33,954 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 09:39:33,954 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-14 09:39:33,954 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:39:33,954 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35311.20 MB 2025-02-14 09:39:33,954 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35978.49 MB 2025-02-14 09:39:33,954 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 667.29 MB 2025-02-14 09:39:33,954 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42886.76 MB 2025-02-14 09:39:33,954 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43245.37 MB 2025-02-14 09:39:33,954 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 358.61 MB 2025-02-14 09:39:33,954 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36594.27 MB 2025-02-14 09:39:33,970 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 09:39:33,970 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 09:39:33,970 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:39:33,970 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:39:33,970 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36337.71 MB 2025-02-14 09:39:33,970 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36555.01 MB 2025-02-14 09:39:33,970 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 217.30 MB 2025-02-14 09:39:33,970 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43245.37 MB 2025-02-14 09:39:33,970 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43245.37 MB 2025-02-14 09:39:33,970 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:39:33,970 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36678.64 MB 2025-02-14 09:39:33,971 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 09:39:33,972 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 09:39:33,972 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.05 seconds 2025-02-14 09:39:33,972 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:39:33,972 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25349.63 MB 2025-02-14 09:39:33,972 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36756.08 MB 2025-02-14 09:39:33,972 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11406.45 MB 2025-02-14 09:39:33,972 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42544.92 MB 2025-02-14 09:39:33,972 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43245.37 MB 2025-02-14 09:39:33,972 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 700.45 MB 2025-02-14 09:39:33,972 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36756.08 MB 2025-02-14 09:39:34,240 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 09:39:34,240 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 09:39:34,240 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 09:39:34,240 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:39:34,240 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36756.08 MB 2025-02-14 09:39:34,240 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39770.11 MB 2025-02-14 09:39:34,240 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-14 09:39:34,240 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43245.37 MB 2025-02-14 09:39:34,240 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43245.37 MB 2025-02-14 09:39:34,240 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:39:34,240 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40071.48 MB 2025-02-14 09:39:34,258 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 09:39:34,258 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 09:39:34,264 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 09:39:34,264 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 09:39:34,264 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:39:34,264 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:39:34,264 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30108.43 MB 2025-02-14 09:39:34,264 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38547.46 MB 2025-02-14 09:39:34,264 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 09:39:34,264 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43245.37 MB 2025-02-14 09:39:34,264 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51636.08 MB 2025-02-14 09:39:34,264 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 09:39:34,264 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38547.46 MB 2025-02-14 09:39:34,427 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 09:39:34,429 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:39:34,429 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 09:39:34,430 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:39:34,430 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 09:39:34,435 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 09:39:34,436 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:39:34,436 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 09:39:34,436 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 09:39:47,693 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:39:47,693 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 09:39:47,698 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 09:39:47,701 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:39:47,701 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 963, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 09:39:47,702 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:39:47,702 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 963, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 09:40:02,652 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 09:40:02,652 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 09:40:02,652 - resource_logging.py:150 - __exit__ - DEBUG - Time: 14.95 seconds 2025-02-14 09:40:02,652 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:40:02,652 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27743.19 MB 2025-02-14 09:40:02,652 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31151.19 MB 2025-02-14 09:40:02,653 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3408.00 MB 2025-02-14 09:40:02,653 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64221.09 MB 2025-02-14 09:40:02,653 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39887.83 MB 2025-02-14 09:40:02,653 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24333.25 MB 2025-02-14 09:40:02,653 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40159.77 MB 2025-02-14 09:40:02,737 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 09:40:02,737 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 09:40:02,737 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 09:40:02,737 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:40:02,737 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31151.19 MB 2025-02-14 09:40:02,737 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28848.33 MB 2025-02-14 09:40:02,737 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2302.86 MB 2025-02-14 09:40:02,737 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39887.83 MB 2025-02-14 09:40:02,737 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48054.14 MB 2025-02-14 09:40:02,737 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8166.31 MB 2025-02-14 09:40:02,737 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42137.33 MB 2025-02-14 09:40:04,650 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 09:40:04,650 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 09:40:04,650 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 09:40:04,650 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:40:04,650 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28848.33 MB 2025-02-14 09:40:04,650 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29379.17 MB 2025-02-14 09:40:04,650 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 09:40:04,650 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48054.14 MB 2025-02-14 09:40:04,650 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37893.44 MB 2025-02-14 09:40:04,650 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10160.70 MB 2025-02-14 09:40:04,650 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33358.51 MB 2025-02-14 09:40:04,663 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 09:40:04,663 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 09:40:04,663 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:40:04,663 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:40:04,663 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29379.17 MB 2025-02-14 09:40:04,663 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31268.53 MB 2025-02-14 09:40:04,663 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.35 MB 2025-02-14 09:40:04,663 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37893.44 MB 2025-02-14 09:40:04,663 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37893.44 MB 2025-02-14 09:40:04,663 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:40:04,663 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32685.96 MB 2025-02-14 09:40:04,872 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 09:40:04,872 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 09:40:04,872 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 09:40:04,872 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:40:04,872 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31268.53 MB 2025-02-14 09:40:04,872 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33510.38 MB 2025-02-14 09:40:04,872 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 09:40:04,872 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37893.44 MB 2025-02-14 09:40:04,872 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42612.03 MB 2025-02-14 09:40:04,872 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 09:40:04,872 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39055.57 MB 2025-02-14 09:40:04,873 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 09:40:04,873 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 09:40:04,873 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 09:40:04,873 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:40:04,873 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29379.17 MB 2025-02-14 09:40:04,873 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33510.38 MB 2025-02-14 09:40:04,873 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.21 MB 2025-02-14 09:40:04,873 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37893.44 MB 2025-02-14 09:40:04,873 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42612.03 MB 2025-02-14 09:40:04,873 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 09:40:04,873 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39055.57 MB 2025-02-14 09:40:05,040 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 09:40:05,040 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 09:40:05,040 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 09:40:05,040 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:40:05,040 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35044.83 MB 2025-02-14 09:40:05,040 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35811.83 MB 2025-02-14 09:40:05,040 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 09:40:05,040 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42612.03 MB 2025-02-14 09:40:05,040 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43025.17 MB 2025-02-14 09:40:05,040 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 09:40:05,040 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36519.62 MB 2025-02-14 09:40:05,059 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 09:40:05,059 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 09:40:05,059 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:40:05,059 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:40:05,059 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36224.72 MB 2025-02-14 09:40:05,059 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36453.51 MB 2025-02-14 09:40:05,059 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.79 MB 2025-02-14 09:40:05,059 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43025.17 MB 2025-02-14 09:40:05,059 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43025.17 MB 2025-02-14 09:40:05,059 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:40:05,059 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36690.94 MB 2025-02-14 09:40:05,060 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 09:40:05,060 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 09:40:05,060 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.36 seconds 2025-02-14 09:40:05,060 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:40:05,060 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24388.02 MB 2025-02-14 09:40:05,060 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36654.21 MB 2025-02-14 09:40:05,060 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12266.19 MB 2025-02-14 09:40:05,060 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64221.09 MB 2025-02-14 09:40:05,060 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43025.17 MB 2025-02-14 09:40:05,060 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21195.92 MB 2025-02-14 09:40:05,060 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36690.94 MB 2025-02-14 09:40:05,328 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 09:40:05,328 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 09:40:05,328 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 09:40:05,329 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:40:05,329 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36654.21 MB 2025-02-14 09:40:05,329 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29387.78 MB 2025-02-14 09:40:05,329 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7266.43 MB 2025-02-14 09:40:05,329 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43025.17 MB 2025-02-14 09:40:05,329 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43025.17 MB 2025-02-14 09:40:05,329 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:40:05,329 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39161.27 MB 2025-02-14 09:40:05,346 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8147, cut from 8149 2025-02-14 09:40:05,346 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 09:40:05,353 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 09:40:05,353 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 09:40:05,353 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:40:05,353 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:40:05,353 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29387.78 MB 2025-02-14 09:40:05,353 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37810.99 MB 2025-02-14 09:40:05,353 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8423.21 MB 2025-02-14 09:40:05,353 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43025.17 MB 2025-02-14 09:40:05,353 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51401.20 MB 2025-02-14 09:40:05,353 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8376.03 MB 2025-02-14 09:40:05,353 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37810.99 MB 2025-02-14 09:40:05,515 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7939] 2025-02-14 09:40:05,516 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:40:05,516 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 09:40:05,517 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:40:05,517 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 09:40:05,522 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 09:40:05,523 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:40:05,523 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 09:40:05,523 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 09:40:52,157 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:40:52,157 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 09:40:52,162 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 09:40:52,165 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:40:52,165 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 290, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 09:40:52,166 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:40:52,166 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 290, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 09:40:56,623 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 09:40:56,623 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 09:40:56,623 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.45 seconds 2025-02-14 09:40:56,623 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:40:56,623 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23053.62 MB 2025-02-14 09:40:56,623 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24079.91 MB 2025-02-14 09:40:56,623 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1026.29 MB 2025-02-14 09:40:56,623 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59777.22 MB 2025-02-14 09:40:56,623 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27724.35 MB 2025-02-14 09:40:56,623 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32052.87 MB 2025-02-14 09:40:56,623 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32978.78 MB 2025-02-14 09:40:56,642 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 09:40:56,642 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 09:40:56,642 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:40:56,642 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:40:56,642 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24079.91 MB 2025-02-14 09:40:56,642 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24571.04 MB 2025-02-14 09:40:56,642 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 491.13 MB 2025-02-14 09:40:56,642 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27724.35 MB 2025-02-14 09:40:56,642 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30779.90 MB 2025-02-14 09:40:56,642 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3055.55 MB 2025-02-14 09:40:56,642 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28147.94 MB 2025-02-14 09:40:58,014 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 09:40:58,014 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 09:40:58,014 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.37 seconds 2025-02-14 09:40:58,014 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:40:58,014 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24571.04 MB 2025-02-14 09:40:58,014 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24954.58 MB 2025-02-14 09:40:58,014 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 383.53 MB 2025-02-14 09:40:58,014 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30779.90 MB 2025-02-14 09:40:58,014 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28728.89 MB 2025-02-14 09:40:58,014 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2051.01 MB 2025-02-14 09:40:58,014 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28912.39 MB 2025-02-14 09:40:58,025 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 09:40:58,025 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 09:40:58,025 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:40:58,025 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:40:58,025 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24954.58 MB 2025-02-14 09:40:58,025 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26319.75 MB 2025-02-14 09:40:58,025 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1365.18 MB 2025-02-14 09:40:58,025 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28728.89 MB 2025-02-14 09:40:58,025 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29412.56 MB 2025-02-14 09:40:58,025 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 683.67 MB 2025-02-14 09:40:58,025 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27343.85 MB 2025-02-14 09:40:58,178 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 09:40:58,178 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 09:40:58,178 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 09:40:58,178 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:40:58,178 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26319.75 MB 2025-02-14 09:40:58,178 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27939.51 MB 2025-02-14 09:40:58,178 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1619.76 MB 2025-02-14 09:40:58,178 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29412.56 MB 2025-02-14 09:40:58,178 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33856.42 MB 2025-02-14 09:40:58,178 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4443.87 MB 2025-02-14 09:40:58,178 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31946.16 MB 2025-02-14 09:40:58,179 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 09:40:58,179 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 09:40:58,179 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 09:40:58,179 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:40:58,179 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24954.58 MB 2025-02-14 09:40:58,179 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27939.51 MB 2025-02-14 09:40:58,179 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2984.94 MB 2025-02-14 09:40:58,179 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28728.89 MB 2025-02-14 09:40:58,179 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33856.42 MB 2025-02-14 09:40:58,179 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5127.54 MB 2025-02-14 09:40:58,179 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31946.16 MB 2025-02-14 09:40:58,301 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 09:40:58,301 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 09:40:58,301 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 09:40:58,301 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:40:58,301 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29047.50 MB 2025-02-14 09:40:58,301 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29602.57 MB 2025-02-14 09:40:58,301 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 555.08 MB 2025-02-14 09:40:58,301 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33856.42 MB 2025-02-14 09:40:58,301 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34150.02 MB 2025-02-14 09:40:58,301 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 293.60 MB 2025-02-14 09:40:58,301 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30113.95 MB 2025-02-14 09:40:58,315 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 09:40:58,315 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 09:40:58,315 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:40:58,315 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:40:58,316 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29900.89 MB 2025-02-14 09:40:58,316 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30131.18 MB 2025-02-14 09:40:58,316 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 230.29 MB 2025-02-14 09:40:58,316 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34150.02 MB 2025-02-14 09:40:58,316 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34152.12 MB 2025-02-14 09:40:58,316 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-14 09:40:58,316 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30234.25 MB 2025-02-14 09:40:58,317 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 09:40:58,317 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 09:40:58,317 - resource_logging.py:150 - __exit__ - DEBUG - Time: 6.15 seconds 2025-02-14 09:40:58,317 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:40:58,317 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22043.23 MB 2025-02-14 09:40:58,317 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30332.26 MB 2025-02-14 09:40:58,317 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8289.02 MB 2025-02-14 09:40:58,317 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59777.22 MB 2025-02-14 09:40:58,317 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34152.12 MB 2025-02-14 09:40:58,317 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25625.10 MB 2025-02-14 09:40:58,317 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30332.26 MB 2025-02-14 09:40:58,585 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 09:40:58,586 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 09:40:58,586 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 09:40:58,586 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:40:58,586 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30332.26 MB 2025-02-14 09:40:58,586 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33346.29 MB 2025-02-14 09:40:58,586 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-14 09:40:58,586 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34152.12 MB 2025-02-14 09:40:58,586 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34957.43 MB 2025-02-14 09:40:58,586 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 805.31 MB 2025-02-14 09:40:58,586 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33647.92 MB 2025-02-14 09:40:58,603 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 09:40:58,604 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 09:40:58,610 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 09:40:58,610 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 09:40:58,610 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:40:58,610 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:40:58,610 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26524.44 MB 2025-02-14 09:40:58,610 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34963.46 MB 2025-02-14 09:40:58,610 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 09:40:58,610 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34957.43 MB 2025-02-14 09:40:58,610 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45447.38 MB 2025-02-14 09:40:58,610 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 09:40:58,610 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34963.46 MB 2025-02-14 09:40:58,775 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 09:40:58,776 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:40:58,776 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 09:40:58,777 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:40:58,777 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 09:40:58,782 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 09:40:58,783 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:40:58,783 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 09:40:58,783 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 09:41:11,357 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:41:11,357 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 09:41:11,362 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 09:41:11,366 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:41:11,366 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 932, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 09:41:11,367 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:41:11,367 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 932, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 09:41:25,685 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 09:41:25,685 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 09:41:25,685 - resource_logging.py:150 - __exit__ - DEBUG - Time: 14.31 seconds 2025-02-14 09:41:25,685 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:41:25,685 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27527.18 MB 2025-02-14 09:41:25,685 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30826.00 MB 2025-02-14 09:41:25,685 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3298.82 MB 2025-02-14 09:41:25,685 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58032.39 MB 2025-02-14 09:41:25,685 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40481.33 MB 2025-02-14 09:41:25,685 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17551.07 MB 2025-02-14 09:41:25,685 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39716.46 MB 2025-02-14 09:41:25,739 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 09:41:25,739 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 09:41:25,739 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-14 09:41:25,739 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:41:25,739 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30826.00 MB 2025-02-14 09:41:25,739 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28687.17 MB 2025-02-14 09:41:25,739 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2138.82 MB 2025-02-14 09:41:25,739 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40481.33 MB 2025-02-14 09:41:25,739 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46418.36 MB 2025-02-14 09:41:25,739 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5937.04 MB 2025-02-14 09:41:25,739 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41104.74 MB 2025-02-14 09:41:27,645 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 09:41:27,645 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 09:41:27,645 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.90 seconds 2025-02-14 09:41:27,645 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:41:27,645 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28687.17 MB 2025-02-14 09:41:27,645 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29218.02 MB 2025-02-14 09:41:27,645 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 09:41:27,645 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46418.36 MB 2025-02-14 09:41:27,645 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38598.08 MB 2025-02-14 09:41:27,645 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7820.28 MB 2025-02-14 09:41:27,645 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33197.35 MB 2025-02-14 09:41:27,658 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 09:41:27,658 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 09:41:27,658 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:41:27,658 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:41:27,658 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29218.02 MB 2025-02-14 09:41:27,658 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31107.37 MB 2025-02-14 09:41:27,658 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.35 MB 2025-02-14 09:41:27,658 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38598.08 MB 2025-02-14 09:41:27,658 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38598.08 MB 2025-02-14 09:41:27,658 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:41:27,658 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32524.80 MB 2025-02-14 09:41:27,881 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 09:41:27,881 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 09:41:27,881 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 09:41:27,881 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:41:27,881 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31107.37 MB 2025-02-14 09:41:27,881 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33349.22 MB 2025-02-14 09:41:27,881 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 09:41:27,881 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38598.08 MB 2025-02-14 09:41:27,881 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43316.67 MB 2025-02-14 09:41:27,881 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 09:41:27,881 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38894.41 MB 2025-02-14 09:41:27,881 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 09:41:27,881 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 09:41:27,881 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.24 seconds 2025-02-14 09:41:27,881 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:41:27,881 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29218.02 MB 2025-02-14 09:41:27,882 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33349.22 MB 2025-02-14 09:41:27,882 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.21 MB 2025-02-14 09:41:27,882 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38598.08 MB 2025-02-14 09:41:27,882 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43316.67 MB 2025-02-14 09:41:27,882 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 09:41:27,882 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38894.41 MB 2025-02-14 09:41:28,048 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 09:41:28,048 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 09:41:28,048 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 09:41:28,048 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:41:28,048 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34883.67 MB 2025-02-14 09:41:28,048 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35650.67 MB 2025-02-14 09:41:28,048 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 09:41:28,048 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43316.67 MB 2025-02-14 09:41:28,048 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43729.81 MB 2025-02-14 09:41:28,048 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 09:41:28,048 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36358.46 MB 2025-02-14 09:41:28,067 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 09:41:28,067 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 09:41:28,067 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:41:28,067 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:41:28,067 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36063.56 MB 2025-02-14 09:41:28,067 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36292.72 MB 2025-02-14 09:41:28,067 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.16 MB 2025-02-14 09:41:28,067 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43729.81 MB 2025-02-14 09:41:28,067 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43729.81 MB 2025-02-14 09:41:28,067 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:41:28,067 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36509.61 MB 2025-02-14 09:41:28,068 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 09:41:28,068 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 09:41:28,068 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.70 seconds 2025-02-14 09:41:28,069 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:41:28,069 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24280.01 MB 2025-02-14 09:41:28,069 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36493.79 MB 2025-02-14 09:41:28,069 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12213.77 MB 2025-02-14 09:41:28,069 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58032.39 MB 2025-02-14 09:41:28,069 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43729.81 MB 2025-02-14 09:41:28,069 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14302.58 MB 2025-02-14 09:41:28,069 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36509.61 MB 2025-02-14 09:41:28,337 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 09:41:28,337 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 09:41:28,337 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 09:41:28,337 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:41:28,337 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36493.79 MB 2025-02-14 09:41:28,337 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29285.30 MB 2025-02-14 09:41:28,337 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7208.48 MB 2025-02-14 09:41:28,337 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43729.81 MB 2025-02-14 09:41:28,337 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43729.81 MB 2025-02-14 09:41:28,337 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:41:28,337 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39005.46 MB 2025-02-14 09:41:28,355 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 09:41:28,356 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 09:41:28,362 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 09:41:28,362 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 09:41:28,362 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:41:28,362 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:41:28,362 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29285.30 MB 2025-02-14 09:41:28,362 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37724.33 MB 2025-02-14 09:41:28,362 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 09:41:28,362 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43729.81 MB 2025-02-14 09:41:28,362 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52120.52 MB 2025-02-14 09:41:28,362 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 09:41:28,362 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37724.33 MB 2025-02-14 09:41:28,524 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 09:41:28,525 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:41:28,525 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 09:41:28,526 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:41:28,526 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 09:41:28,531 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 09:41:28,532 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:41:28,532 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 09:41:28,532 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 09:41:37,241 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:41:37,241 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 09:41:37,247 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 09:41:37,250 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:41:37,250 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 234, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 09:41:37,251 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:41:37,251 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 234, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 09:41:40,868 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 09:41:40,868 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 09:41:40,868 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.61 seconds 2025-02-14 09:41:40,868 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:41:40,868 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22664.05 MB 2025-02-14 09:41:40,868 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23492.16 MB 2025-02-14 09:41:40,868 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 828.11 MB 2025-02-14 09:41:40,868 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64705.53 MB 2025-02-14 09:41:40,868 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26696.74 MB 2025-02-14 09:41:40,868 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -38008.78 MB 2025-02-14 09:41:40,868 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32362.72 MB 2025-02-14 09:41:40,885 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 09:41:40,885 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 09:41:40,885 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:41:40,885 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:41:40,885 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23492.16 MB 2025-02-14 09:41:40,885 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23893.58 MB 2025-02-14 09:41:40,885 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 401.42 MB 2025-02-14 09:41:40,885 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26696.74 MB 2025-02-14 09:41:40,885 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28756.15 MB 2025-02-14 09:41:40,885 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2059.40 MB 2025-02-14 09:41:40,885 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26814.85 MB 2025-02-14 09:41:42,002 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 09:41:42,002 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 09:41:42,002 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.11 seconds 2025-02-14 09:41:42,002 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:41:42,002 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23893.58 MB 2025-02-14 09:41:42,002 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24204.12 MB 2025-02-14 09:41:42,002 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 310.54 MB 2025-02-14 09:41:42,002 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28756.15 MB 2025-02-14 09:41:42,002 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27927.77 MB 2025-02-14 09:41:42,002 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -828.38 MB 2025-02-14 09:41:42,002 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28151.03 MB 2025-02-14 09:41:42,011 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 09:41:42,011 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 09:41:42,011 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:41:42,011 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:41:42,011 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24204.12 MB 2025-02-14 09:41:42,011 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25309.23 MB 2025-02-14 09:41:42,011 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1105.11 MB 2025-02-14 09:41:42,011 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27927.77 MB 2025-02-14 09:41:42,011 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27927.77 MB 2025-02-14 09:41:42,011 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:41:42,011 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26138.43 MB 2025-02-14 09:41:42,135 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 09:41:42,135 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 09:41:42,135 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 09:41:42,135 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:41:42,135 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25309.23 MB 2025-02-14 09:41:42,135 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26620.74 MB 2025-02-14 09:41:42,135 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1311.51 MB 2025-02-14 09:41:42,135 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27927.77 MB 2025-02-14 09:41:42,135 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31803.31 MB 2025-02-14 09:41:42,135 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3875.54 MB 2025-02-14 09:41:42,135 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29865.69 MB 2025-02-14 09:41:42,136 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 09:41:42,136 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 09:41:42,136 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 09:41:42,136 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:41:42,136 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24204.12 MB 2025-02-14 09:41:42,136 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26620.74 MB 2025-02-14 09:41:42,136 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2416.62 MB 2025-02-14 09:41:42,136 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27927.77 MB 2025-02-14 09:41:42,136 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31803.31 MB 2025-02-14 09:41:42,136 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3875.54 MB 2025-02-14 09:41:42,136 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29865.69 MB 2025-02-14 09:41:42,234 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 09:41:42,234 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 09:41:42,234 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 09:41:42,234 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:41:42,234 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27517.86 MB 2025-02-14 09:41:42,234 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27967.35 MB 2025-02-14 09:41:42,234 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 449.48 MB 2025-02-14 09:41:42,234 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31803.31 MB 2025-02-14 09:41:42,234 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32040.29 MB 2025-02-14 09:41:42,234 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 236.98 MB 2025-02-14 09:41:42,234 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28381.40 MB 2025-02-14 09:41:42,246 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 09:41:42,246 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 09:41:42,246 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:41:42,246 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:41:42,246 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28208.89 MB 2025-02-14 09:41:42,246 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28413.91 MB 2025-02-14 09:41:42,246 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.02 MB 2025-02-14 09:41:42,247 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32040.29 MB 2025-02-14 09:41:42,247 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32044.48 MB 2025-02-14 09:41:42,247 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-14 09:41:42,247 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28483.44 MB 2025-02-14 09:41:42,248 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 09:41:42,248 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 09:41:42,248 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.99 seconds 2025-02-14 09:41:42,248 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:41:42,248 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21848.77 MB 2025-02-14 09:41:42,248 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28614.98 MB 2025-02-14 09:41:42,248 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6766.21 MB 2025-02-14 09:41:42,248 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64705.53 MB 2025-02-14 09:41:42,248 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32044.48 MB 2025-02-14 09:41:42,248 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32661.05 MB 2025-02-14 09:41:42,248 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28614.98 MB 2025-02-14 09:41:42,518 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 09:41:42,518 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 09:41:42,518 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 09:41:42,518 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:41:42,518 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28614.98 MB 2025-02-14 09:41:42,518 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31629.02 MB 2025-02-14 09:41:42,518 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-14 09:41:42,518 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32044.48 MB 2025-02-14 09:41:42,518 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33386.66 MB 2025-02-14 09:41:42,518 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1342.18 MB 2025-02-14 09:41:42,518 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31930.65 MB 2025-02-14 09:41:42,536 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 09:41:42,536 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 09:41:42,542 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 09:41:42,542 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 09:41:42,542 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:41:42,542 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:41:42,542 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26070.02 MB 2025-02-14 09:41:42,542 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34509.05 MB 2025-02-14 09:41:42,542 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 09:41:42,542 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33386.66 MB 2025-02-14 09:41:42,542 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43876.61 MB 2025-02-14 09:41:42,542 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 09:41:42,542 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34509.05 MB 2025-02-14 09:41:42,704 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 09:41:42,706 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:41:42,706 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 09:41:42,707 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:41:42,707 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 09:41:42,712 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 09:41:42,713 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:41:42,713 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 09:41:42,713 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 09:41:50,576 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:41:50,576 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 09:41:50,581 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 09:41:50,585 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:41:50,585 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 136, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 09:41:50,586 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:41:50,586 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 136, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 09:41:52,713 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 09:41:52,713 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 09:41:52,713 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.12 seconds 2025-02-14 09:41:52,713 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:41:52,713 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21980.52 MB 2025-02-14 09:41:52,713 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22461.82 MB 2025-02-14 09:41:52,713 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 481.30 MB 2025-02-14 09:41:52,713 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56461.62 MB 2025-02-14 09:41:52,713 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27801.94 MB 2025-02-14 09:41:52,713 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28659.68 MB 2025-02-14 09:41:52,713 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31452.70 MB 2025-02-14 09:41:52,723 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 09:41:52,723 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 09:41:52,723 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:41:52,723 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:41:52,723 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22461.82 MB 2025-02-14 09:41:52,723 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22652.87 MB 2025-02-14 09:41:52,723 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 191.05 MB 2025-02-14 09:41:52,723 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27801.94 MB 2025-02-14 09:41:52,723 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27801.94 MB 2025-02-14 09:41:52,723 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:41:52,723 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24316.18 MB 2025-02-14 09:41:53,353 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 09:41:53,353 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 09:41:53,353 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.63 seconds 2025-02-14 09:41:53,353 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:41:53,353 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22652.87 MB 2025-02-14 09:41:53,353 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22825.39 MB 2025-02-14 09:41:53,353 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 172.52 MB 2025-02-14 09:41:53,353 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27801.94 MB 2025-02-14 09:41:53,354 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27525.12 MB 2025-02-14 09:41:53,354 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -276.82 MB 2025-02-14 09:41:53,354 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26824.34 MB 2025-02-14 09:41:53,360 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 09:41:53,360 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 09:41:53,360 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 09:41:53,360 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:41:53,360 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22825.32 MB 2025-02-14 09:41:53,360 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23439.27 MB 2025-02-14 09:41:53,360 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 613.95 MB 2025-02-14 09:41:53,360 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27525.12 MB 2025-02-14 09:41:53,360 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27525.12 MB 2025-02-14 09:41:53,360 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:41:53,360 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23899.94 MB 2025-02-14 09:41:53,430 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 09:41:53,430 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 09:41:53,431 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 09:41:53,431 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:41:53,431 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23439.27 MB 2025-02-14 09:41:53,431 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24167.92 MB 2025-02-14 09:41:53,431 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 728.65 MB 2025-02-14 09:41:53,431 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27525.12 MB 2025-02-14 09:41:53,431 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27525.12 MB 2025-02-14 09:41:53,431 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:41:53,431 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25969.77 MB 2025-02-14 09:41:53,431 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 09:41:53,431 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 09:41:53,431 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 09:41:53,431 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:41:53,431 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22825.32 MB 2025-02-14 09:41:53,431 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24167.92 MB 2025-02-14 09:41:53,431 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1342.60 MB 2025-02-14 09:41:53,431 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27525.12 MB 2025-02-14 09:41:53,431 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27525.12 MB 2025-02-14 09:41:53,431 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:41:53,431 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25969.77 MB 2025-02-14 09:41:53,487 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 09:41:53,487 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 09:41:53,487 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-14 09:41:53,487 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:41:53,487 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24666.32 MB 2025-02-14 09:41:53,487 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24915.60 MB 2025-02-14 09:41:53,487 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 249.28 MB 2025-02-14 09:41:53,487 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27525.12 MB 2025-02-14 09:41:53,487 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27655.14 MB 2025-02-14 09:41:53,487 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 130.02 MB 2025-02-14 09:41:53,487 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25157.19 MB 2025-02-14 09:41:53,496 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 09:41:53,496 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 09:41:53,496 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:41:53,496 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:41:53,496 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25049.80 MB 2025-02-14 09:41:53,496 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25254.75 MB 2025-02-14 09:41:53,496 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 204.95 MB 2025-02-14 09:41:53,496 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27655.14 MB 2025-02-14 09:41:53,496 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27659.34 MB 2025-02-14 09:41:53,496 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-14 09:41:53,496 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25254.75 MB 2025-02-14 09:41:53,497 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 09:41:53,497 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 09:41:53,497 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.91 seconds 2025-02-14 09:41:53,497 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:41:53,497 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21506.69 MB 2025-02-14 09:41:53,497 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25455.55 MB 2025-02-14 09:41:53,497 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3948.87 MB 2025-02-14 09:41:53,497 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56461.62 MB 2025-02-14 09:41:53,497 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27659.34 MB 2025-02-14 09:41:53,497 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28802.29 MB 2025-02-14 09:41:53,497 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25455.55 MB 2025-02-14 09:41:53,766 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 09:41:53,766 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 09:41:53,766 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 09:41:53,766 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:41:53,766 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25455.55 MB 2025-02-14 09:41:53,766 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25232.90 MB 2025-02-14 09:41:53,766 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -222.65 MB 2025-02-14 09:41:53,766 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27659.34 MB 2025-02-14 09:41:53,766 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27659.34 MB 2025-02-14 09:41:53,766 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:41:53,766 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26860.19 MB 2025-02-14 09:41:53,784 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8151, cut from 8153 2025-02-14 09:41:53,784 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 09:41:53,790 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 09:41:53,790 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 09:41:53,790 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:41:53,790 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:41:53,790 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25232.90 MB 2025-02-14 09:41:53,790 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33660.24 MB 2025-02-14 09:41:53,790 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8427.34 MB 2025-02-14 09:41:53,790 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27659.34 MB 2025-02-14 09:41:53,790 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38134.61 MB 2025-02-14 09:41:53,790 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10475.27 MB 2025-02-14 09:41:53,790 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33660.24 MB 2025-02-14 09:41:53,951 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7943] 2025-02-14 09:41:53,952 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:41:53,952 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 09:41:53,953 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:41:53,953 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 09:41:53,958 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 09:41:53,959 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:41:53,959 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 09:41:53,959 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 09:42:06,153 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:42:06,153 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 09:42:06,158 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 09:42:06,161 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:42:06,161 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 128, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 09:42:06,162 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:42:06,162 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 128, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 09:42:08,133 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 09:42:08,133 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 09:42:08,133 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.97 seconds 2025-02-14 09:42:08,133 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:42:08,133 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21924.78 MB 2025-02-14 09:42:08,133 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22377.76 MB 2025-02-14 09:42:08,133 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 452.98 MB 2025-02-14 09:42:08,133 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46514.83 MB 2025-02-14 09:42:08,133 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26971.47 MB 2025-02-14 09:42:08,133 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19543.36 MB 2025-02-14 09:42:08,133 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31396.95 MB 2025-02-14 09:42:08,141 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 09:42:08,141 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 09:42:08,141 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:42:08,141 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:42:08,141 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22377.76 MB 2025-02-14 09:42:08,141 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22316.31 MB 2025-02-14 09:42:08,141 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -61.45 MB 2025-02-14 09:42:08,141 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26971.47 MB 2025-02-14 09:42:08,141 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26971.47 MB 2025-02-14 09:42:08,141 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:42:08,141 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23671.29 MB 2025-02-14 09:42:08,570 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 09:42:08,570 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 09:42:08,570 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.43 seconds 2025-02-14 09:42:08,570 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:42:08,570 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22316.31 MB 2025-02-14 09:42:08,570 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22433.10 MB 2025-02-14 09:42:08,570 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 116.79 MB 2025-02-14 09:42:08,570 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26971.47 MB 2025-02-14 09:42:08,570 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26694.65 MB 2025-02-14 09:42:08,570 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -276.82 MB 2025-02-14 09:42:08,570 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26402.85 MB 2025-02-14 09:42:08,575 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 09:42:08,575 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 09:42:08,575 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 09:42:08,575 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:42:08,575 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22433.03 MB 2025-02-14 09:42:08,575 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22848.63 MB 2025-02-14 09:42:08,576 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 415.60 MB 2025-02-14 09:42:08,576 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26694.65 MB 2025-02-14 09:42:08,576 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26694.65 MB 2025-02-14 09:42:08,576 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:42:08,576 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23160.47 MB 2025-02-14 09:42:08,664 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 09:42:08,664 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 09:42:08,664 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 09:42:08,664 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:42:08,664 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22848.63 MB 2025-02-14 09:42:08,664 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23353.42 MB 2025-02-14 09:42:08,664 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 504.79 MB 2025-02-14 09:42:08,664 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26694.65 MB 2025-02-14 09:42:08,664 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26694.65 MB 2025-02-14 09:42:08,664 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:42:08,664 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24561.58 MB 2025-02-14 09:42:08,664 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 09:42:08,664 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 09:42:08,664 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 09:42:08,665 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:42:08,665 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22433.03 MB 2025-02-14 09:42:08,665 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23353.42 MB 2025-02-14 09:42:08,665 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 920.39 MB 2025-02-14 09:42:08,665 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26694.65 MB 2025-02-14 09:42:08,665 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26694.65 MB 2025-02-14 09:42:08,665 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:42:08,665 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24561.58 MB 2025-02-14 09:42:08,709 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 09:42:08,709 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 09:42:08,709 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-14 09:42:08,709 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:42:08,709 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23840.74 MB 2025-02-14 09:42:08,709 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24052.74 MB 2025-02-14 09:42:08,709 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 211.99 MB 2025-02-14 09:42:08,709 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26694.65 MB 2025-02-14 09:42:08,709 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26826.77 MB 2025-02-14 09:42:08,709 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 132.12 MB 2025-02-14 09:42:08,709 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24208.45 MB 2025-02-14 09:42:08,715 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 09:42:08,715 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 09:42:08,715 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 09:42:08,715 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:42:08,715 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24187.62 MB 2025-02-14 09:42:08,715 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24398.90 MB 2025-02-14 09:42:08,715 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 211.28 MB 2025-02-14 09:42:08,715 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26826.77 MB 2025-02-14 09:42:08,715 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26826.77 MB 2025-02-14 09:42:08,715 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:42:08,715 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24398.90 MB 2025-02-14 09:42:08,716 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 09:42:08,716 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 09:42:08,716 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.55 seconds 2025-02-14 09:42:08,716 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:42:08,716 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21478.81 MB 2025-02-14 09:42:08,716 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24586.52 MB 2025-02-14 09:42:08,716 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3107.71 MB 2025-02-14 09:42:08,716 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46514.83 MB 2025-02-14 09:42:08,716 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26826.77 MB 2025-02-14 09:42:08,716 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19688.06 MB 2025-02-14 09:42:08,716 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24586.52 MB 2025-02-14 09:42:08,965 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 09:42:08,965 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 09:42:08,965 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.25 seconds 2025-02-14 09:42:08,965 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:42:08,965 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21989.83 MB 2025-02-14 09:42:08,965 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24802.21 MB 2025-02-14 09:42:08,965 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2812.39 MB 2025-02-14 09:42:08,965 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26826.77 MB 2025-02-14 09:42:08,965 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26826.77 MB 2025-02-14 09:42:08,965 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:42:08,965 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25083.42 MB 2025-02-14 09:42:08,982 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 7615, cut from 7617 2025-02-14 09:42:08,982 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 09:42:08,988 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 09:42:08,988 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 09:42:08,988 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:42:08,988 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:42:08,988 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24802.21 MB 2025-02-14 09:42:08,988 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32676.29 MB 2025-02-14 09:42:08,988 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7874.08 MB 2025-02-14 09:42:08,988 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26826.77 MB 2025-02-14 09:42:08,988 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36616.27 MB 2025-02-14 09:42:08,988 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9789.51 MB 2025-02-14 09:42:08,988 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32676.29 MB 2025-02-14 09:42:09,139 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7407] 2025-02-14 09:42:09,140 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:42:09,140 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 09:42:09,141 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:42:09,141 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 09:42:09,146 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 09:42:09,147 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:42:09,147 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 09:42:09,147 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 09:43:25,914 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:43:25,914 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 09:43:25,919 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 09:43:25,922 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:43:25,922 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 233, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 09:43:25,923 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:43:25,923 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 233, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 09:43:29,479 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 09:43:29,479 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 09:43:29,479 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.55 seconds 2025-02-14 09:43:29,479 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:43:29,479 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22656.43 MB 2025-02-14 09:43:29,479 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23481.01 MB 2025-02-14 09:43:29,479 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 824.57 MB 2025-02-14 09:43:29,479 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44447.04 MB 2025-02-14 09:43:29,479 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26694.65 MB 2025-02-14 09:43:29,479 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17752.39 MB 2025-02-14 09:43:29,479 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32355.10 MB 2025-02-14 09:43:29,488 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 09:43:29,488 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 09:43:29,488 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:43:29,488 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:43:29,488 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23481.01 MB 2025-02-14 09:43:29,488 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22350.45 MB 2025-02-14 09:43:29,488 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1130.56 MB 2025-02-14 09:43:29,488 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26694.65 MB 2025-02-14 09:43:29,488 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26694.65 MB 2025-02-14 09:43:29,488 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:43:29,488 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23724.57 MB 2025-02-14 09:43:29,571 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 09:43:29,571 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 09:43:29,571 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 09:43:29,571 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:43:29,571 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22350.45 MB 2025-02-14 09:43:29,571 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22370.36 MB 2025-02-14 09:43:29,571 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 19.91 MB 2025-02-14 09:43:29,571 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26694.65 MB 2025-02-14 09:43:29,571 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26694.65 MB 2025-02-14 09:43:29,571 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:43:29,571 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23309.33 MB 2025-02-14 09:43:29,575 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 09:43:29,575 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 09:43:29,575 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 09:43:29,575 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:43:29,575 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22370.29 MB 2025-02-14 09:43:29,575 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22441.13 MB 2025-02-14 09:43:29,575 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 70.84 MB 2025-02-14 09:43:29,575 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26694.65 MB 2025-02-14 09:43:29,575 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26694.65 MB 2025-02-14 09:43:29,575 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:43:29,575 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22495.28 MB 2025-02-14 09:43:29,589 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 09:43:29,589 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 09:43:29,589 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:43:29,589 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:43:29,589 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22441.13 MB 2025-02-14 09:43:29,589 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22526.24 MB 2025-02-14 09:43:29,589 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 85.11 MB 2025-02-14 09:43:29,589 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26694.65 MB 2025-02-14 09:43:29,589 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26694.65 MB 2025-02-14 09:43:29,589 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:43:29,589 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22734.09 MB 2025-02-14 09:43:29,589 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 09:43:29,589 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 09:43:29,589 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:43:29,589 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:43:29,589 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22370.29 MB 2025-02-14 09:43:29,589 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22526.24 MB 2025-02-14 09:43:29,589 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 155.95 MB 2025-02-14 09:43:29,589 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26694.65 MB 2025-02-14 09:43:29,590 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26694.65 MB 2025-02-14 09:43:29,590 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:43:29,590 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22734.09 MB 2025-02-14 09:43:29,599 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 09:43:29,599 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 09:43:29,599 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:43:29,599 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:43:29,599 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22583.75 MB 2025-02-14 09:43:29,599 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22612.52 MB 2025-02-14 09:43:29,599 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 28.76 MB 2025-02-14 09:43:29,599 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26694.65 MB 2025-02-14 09:43:29,599 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26703.04 MB 2025-02-14 09:43:29,599 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8.39 MB 2025-02-14 09:43:29,599 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22649.27 MB 2025-02-14 09:43:29,601 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 09:43:29,601 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 09:43:29,601 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 09:43:29,601 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:43:29,601 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22628.77 MB 2025-02-14 09:43:29,601 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22648.66 MB 2025-02-14 09:43:29,601 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 19.90 MB 2025-02-14 09:43:29,602 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26703.04 MB 2025-02-14 09:43:29,602 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26703.04 MB 2025-02-14 09:43:29,602 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:43:29,602 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22648.66 MB 2025-02-14 09:43:29,602 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 09:43:29,602 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 09:43:29,603 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.68 seconds 2025-02-14 09:43:29,603 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:43:29,603 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21844.64 MB 2025-02-14 09:43:29,603 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22685.47 MB 2025-02-14 09:43:29,603 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 840.83 MB 2025-02-14 09:43:29,603 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44447.04 MB 2025-02-14 09:43:29,603 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26703.04 MB 2025-02-14 09:43:29,603 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17744.00 MB 2025-02-14 09:43:29,603 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22685.47 MB 2025-02-14 09:43:29,666 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 09:43:29,666 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 09:43:29,666 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 09:43:29,666 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:43:29,666 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22685.47 MB 2025-02-14 09:43:29,666 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23238.28 MB 2025-02-14 09:43:29,666 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 552.81 MB 2025-02-14 09:43:29,666 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26703.04 MB 2025-02-14 09:43:29,666 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26707.23 MB 2025-02-14 09:43:29,666 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-14 09:43:29,666 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23293.46 MB 2025-02-14 09:43:29,671 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 1483, cut from 1485 2025-02-14 09:43:29,671 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-14 09:43:29,673 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 09:43:29,673 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 09:43:29,673 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 09:43:29,673 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:43:29,673 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22488.13 MB 2025-02-14 09:43:29,673 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24032.81 MB 2025-02-14 09:43:29,673 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1544.68 MB 2025-02-14 09:43:29,673 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26707.23 MB 2025-02-14 09:43:29,673 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26707.23 MB 2025-02-14 09:43:29,673 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:43:29,673 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24032.81 MB 2025-02-14 09:43:29,702 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 1275] 2025-02-14 09:43:29,704 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:43:29,704 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 09:43:29,705 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:43:29,705 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 09:43:29,710 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 09:43:29,711 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:43:29,711 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 09:43:29,711 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-14 09:44:35,089 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:44:35,089 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 09:44:35,094 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 09:44:35,098 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:44:35,098 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1780, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 09:44:35,099 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:44:35,099 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1780, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 09:45:02,309 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 09:45:02,309 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 09:45:02,309 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.20 seconds 2025-02-14 09:45:02,309 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:45:02,309 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33436.70 MB 2025-02-14 09:45:02,309 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39736.55 MB 2025-02-14 09:45:02,309 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6299.84 MB 2025-02-14 09:45:02,309 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41437.63 MB 2025-02-14 09:45:02,309 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45401.24 MB 2025-02-14 09:45:02,309 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3963.62 MB 2025-02-14 09:45:02,309 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48571.19 MB 2025-02-14 09:45:02,421 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 09:45:02,421 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 09:45:02,421 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 09:45:02,421 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:45:02,421 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39736.55 MB 2025-02-14 09:45:02,421 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33096.97 MB 2025-02-14 09:45:02,421 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6639.57 MB 2025-02-14 09:45:02,421 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45401.24 MB 2025-02-14 09:45:02,421 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 66764.93 MB 2025-02-14 09:45:02,421 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 21363.69 MB 2025-02-14 09:45:02,421 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 58129.67 MB 2025-02-14 09:45:04,337 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 09:45:04,337 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 09:45:04,337 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 09:45:04,337 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:45:04,337 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33096.97 MB 2025-02-14 09:45:04,337 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33627.81 MB 2025-02-14 09:45:04,337 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 09:45:04,337 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 66764.93 MB 2025-02-14 09:45:04,337 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38075.89 MB 2025-02-14 09:45:04,337 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28689.04 MB 2025-02-14 09:45:04,337 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37607.15 MB 2025-02-14 09:45:04,351 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 09:45:04,351 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 09:45:04,351 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:45:04,351 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:45:04,351 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33627.81 MB 2025-02-14 09:45:04,351 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35517.17 MB 2025-02-14 09:45:04,351 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.35 MB 2025-02-14 09:45:04,351 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38075.89 MB 2025-02-14 09:45:04,351 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39963.33 MB 2025-02-14 09:45:04,351 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 09:45:04,351 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36934.60 MB 2025-02-14 09:45:04,561 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 09:45:04,561 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 09:45:04,561 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 09:45:04,562 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:45:04,562 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35517.17 MB 2025-02-14 09:45:04,562 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37759.02 MB 2025-02-14 09:45:04,562 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 09:45:04,562 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39963.33 MB 2025-02-14 09:45:04,562 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46097.50 MB 2025-02-14 09:45:04,562 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 09:45:04,562 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43304.20 MB 2025-02-14 09:45:04,562 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 09:45:04,562 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 09:45:04,562 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 09:45:04,562 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:45:04,562 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33627.81 MB 2025-02-14 09:45:04,562 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37759.02 MB 2025-02-14 09:45:04,562 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.21 MB 2025-02-14 09:45:04,562 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38075.89 MB 2025-02-14 09:45:04,562 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46097.50 MB 2025-02-14 09:45:04,562 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-14 09:45:04,562 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43304.20 MB 2025-02-14 09:45:04,728 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 09:45:04,728 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 09:45:04,728 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 09:45:04,728 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:45:04,728 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39293.47 MB 2025-02-14 09:45:04,728 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40060.47 MB 2025-02-14 09:45:04,728 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 09:45:04,729 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46097.50 MB 2025-02-14 09:45:04,729 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46510.64 MB 2025-02-14 09:45:04,729 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 09:45:04,729 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40768.26 MB 2025-02-14 09:45:04,747 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 09:45:04,747 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 09:45:04,747 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:45:04,747 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:45:04,747 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40473.36 MB 2025-02-14 09:45:04,747 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40702.10 MB 2025-02-14 09:45:04,747 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.74 MB 2025-02-14 09:45:04,747 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46510.64 MB 2025-02-14 09:45:04,747 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46510.64 MB 2025-02-14 09:45:04,747 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:45:04,747 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40909.19 MB 2025-02-14 09:45:04,749 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 09:45:04,749 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 09:45:04,749 - resource_logging.py:150 - __exit__ - DEBUG - Time: 29.65 seconds 2025-02-14 09:45:04,749 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:45:04,749 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27234.78 MB 2025-02-14 09:45:04,749 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40902.75 MB 2025-02-14 09:45:04,749 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13667.97 MB 2025-02-14 09:45:04,749 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35234.25 MB 2025-02-14 09:45:04,749 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46510.64 MB 2025-02-14 09:45:04,749 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 11276.39 MB 2025-02-14 09:45:04,749 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40909.19 MB 2025-02-14 09:45:05,015 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 09:45:05,015 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 09:45:05,015 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 09:45:05,015 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:45:05,015 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40902.75 MB 2025-02-14 09:45:05,015 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32234.63 MB 2025-02-14 09:45:05,015 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8668.12 MB 2025-02-14 09:45:05,015 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46510.64 MB 2025-02-14 09:45:05,015 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46510.64 MB 2025-02-14 09:45:05,015 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:45:05,015 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43409.20 MB 2025-02-14 09:45:05,033 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8145, cut from 8147 2025-02-14 09:45:05,033 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 09:45:05,039 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 09:45:05,039 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 09:45:05,039 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:45:05,039 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:45:05,039 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32234.63 MB 2025-02-14 09:45:05,039 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40656.60 MB 2025-02-14 09:45:05,039 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8421.96 MB 2025-02-14 09:45:05,039 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46510.64 MB 2025-02-14 09:45:05,039 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54882.47 MB 2025-02-14 09:45:05,039 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8371.83 MB 2025-02-14 09:45:05,039 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40656.60 MB 2025-02-14 09:45:05,204 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7937] 2025-02-14 09:45:05,205 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:45:05,205 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 09:45:05,206 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:45:05,206 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 09:45:05,211 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 09:45:05,212 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:45:05,212 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 09:45:05,212 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 09:45:13,199 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:45:13,199 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 09:45:13,207 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 09:45:13,213 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:45:13,214 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1446, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 09:45:13,216 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:45:13,216 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1446, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 09:45:35,666 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 09:45:35,666 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 09:45:35,666 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.44 seconds 2025-02-14 09:45:35,666 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:45:35,666 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31108.81 MB 2025-02-14 09:45:35,666 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36226.12 MB 2025-02-14 09:45:35,666 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5117.31 MB 2025-02-14 09:45:35,666 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63254.30 MB 2025-02-14 09:45:35,666 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45256.54 MB 2025-02-14 09:45:35,666 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17997.76 MB 2025-02-14 09:45:35,666 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45110.03 MB 2025-02-14 09:45:35,747 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 09:45:35,747 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 09:45:35,747 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 09:45:35,747 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:45:35,747 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36226.12 MB 2025-02-14 09:45:35,747 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31359.30 MB 2025-02-14 09:45:35,747 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4866.83 MB 2025-02-14 09:45:35,747 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45256.54 MB 2025-02-14 09:45:35,747 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57378.08 MB 2025-02-14 09:45:35,747 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 12121.54 MB 2025-02-14 09:45:35,748 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50920.67 MB 2025-02-14 09:45:37,670 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 09:45:37,670 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 09:45:37,670 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 09:45:37,670 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:45:37,670 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31359.30 MB 2025-02-14 09:45:37,670 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31890.14 MB 2025-02-14 09:45:37,670 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 09:45:37,670 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57378.08 MB 2025-02-14 09:45:37,670 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38499.52 MB 2025-02-14 09:45:37,670 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18878.56 MB 2025-02-14 09:45:37,670 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35869.47 MB 2025-02-14 09:45:37,685 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 09:45:37,685 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 09:45:37,685 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:45:37,685 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:45:37,685 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31890.14 MB 2025-02-14 09:45:37,685 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33779.49 MB 2025-02-14 09:45:37,685 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.35 MB 2025-02-14 09:45:37,685 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38499.52 MB 2025-02-14 09:45:37,685 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39443.23 MB 2025-02-14 09:45:37,685 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 09:45:37,685 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35196.92 MB 2025-02-14 09:45:37,897 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 09:45:37,897 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 09:45:37,897 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 09:45:37,897 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:45:37,897 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33779.49 MB 2025-02-14 09:45:37,897 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36021.35 MB 2025-02-14 09:45:37,897 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 09:45:37,897 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39443.23 MB 2025-02-14 09:45:37,897 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45105.55 MB 2025-02-14 09:45:37,897 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 09:45:37,897 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41566.53 MB 2025-02-14 09:45:37,897 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 09:45:37,897 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 09:45:37,898 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 09:45:37,898 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:45:37,898 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31890.14 MB 2025-02-14 09:45:37,898 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36021.35 MB 2025-02-14 09:45:37,898 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.21 MB 2025-02-14 09:45:37,898 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38499.52 MB 2025-02-14 09:45:37,898 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45105.55 MB 2025-02-14 09:45:37,898 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 09:45:37,898 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41566.53 MB 2025-02-14 09:45:38,064 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 09:45:38,064 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 09:45:38,064 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 09:45:38,065 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:45:38,065 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37555.79 MB 2025-02-14 09:45:38,065 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38322.80 MB 2025-02-14 09:45:38,065 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 09:45:38,065 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45105.55 MB 2025-02-14 09:45:38,065 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45518.68 MB 2025-02-14 09:45:38,065 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 09:45:38,065 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39030.58 MB 2025-02-14 09:45:38,083 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 09:45:38,083 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 09:45:38,083 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:45:38,083 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:45:38,083 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38735.68 MB 2025-02-14 09:45:38,083 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38964.08 MB 2025-02-14 09:45:38,083 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.40 MB 2025-02-14 09:45:38,083 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45518.68 MB 2025-02-14 09:45:38,083 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45518.68 MB 2025-02-14 09:45:38,083 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:45:38,083 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39182.26 MB 2025-02-14 09:45:38,085 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 09:45:38,085 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 09:45:38,085 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.87 seconds 2025-02-14 09:45:38,085 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:45:38,085 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26070.83 MB 2025-02-14 09:45:38,085 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39164.61 MB 2025-02-14 09:45:38,085 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13093.78 MB 2025-02-14 09:45:38,085 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63254.30 MB 2025-02-14 09:45:38,085 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45518.68 MB 2025-02-14 09:45:38,085 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17735.61 MB 2025-02-14 09:45:38,085 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39182.26 MB 2025-02-14 09:45:38,355 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 09:45:38,355 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 09:45:38,355 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 09:45:38,355 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:45:38,355 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39164.61 MB 2025-02-14 09:45:38,355 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31068.01 MB 2025-02-14 09:45:38,355 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8096.60 MB 2025-02-14 09:45:38,355 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45518.68 MB 2025-02-14 09:45:38,355 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45518.68 MB 2025-02-14 09:45:38,355 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:45:38,355 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41669.52 MB 2025-02-14 09:45:38,373 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8140, cut from 8142 2025-02-14 09:45:38,373 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-14 09:45:38,379 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 09:45:38,379 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 09:45:38,379 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:45:38,379 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:45:38,379 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31068.01 MB 2025-02-14 09:45:38,379 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39484.61 MB 2025-02-14 09:45:38,379 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8416.60 MB 2025-02-14 09:45:38,379 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45518.68 MB 2025-02-14 09:45:38,379 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53886.32 MB 2025-02-14 09:45:38,379 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8367.64 MB 2025-02-14 09:45:38,379 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39484.61 MB 2025-02-14 09:45:38,543 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7932] 2025-02-14 09:45:38,544 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:45:38,544 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 09:45:38,545 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:45:38,545 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 09:45:38,550 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 09:45:38,551 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:45:38,551 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 09:45:38,551 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-14 09:46:42,570 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:46:42,571 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 09:46:42,576 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 09:46:42,579 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:46:42,580 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 109, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 09:46:42,580 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:46:42,580 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 109, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 09:46:44,275 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 09:46:44,275 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 09:46:44,275 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.69 seconds 2025-02-14 09:46:44,275 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:46:44,275 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21792.38 MB 2025-02-14 09:46:44,275 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22178.13 MB 2025-02-14 09:46:44,275 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 385.74 MB 2025-02-14 09:46:44,275 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62253.96 MB 2025-02-14 09:46:44,275 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27638.37 MB 2025-02-14 09:46:44,275 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34615.59 MB 2025-02-14 09:46:44,275 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31037.26 MB 2025-02-14 09:46:44,278 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 09:46:44,278 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 09:46:44,278 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 09:46:44,278 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:46:44,278 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22178.13 MB 2025-02-14 09:46:44,278 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22365.02 MB 2025-02-14 09:46:44,278 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 186.89 MB 2025-02-14 09:46:44,278 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27638.37 MB 2025-02-14 09:46:44,278 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27638.37 MB 2025-02-14 09:46:44,278 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:46:44,278 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22943.70 MB 2025-02-14 09:46:44,798 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 09:46:44,798 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 09:46:44,798 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.52 seconds 2025-02-14 09:46:44,798 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:46:44,798 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22365.02 MB 2025-02-14 09:46:44,798 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22509.67 MB 2025-02-14 09:46:44,798 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 144.65 MB 2025-02-14 09:46:44,798 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27638.37 MB 2025-02-14 09:46:44,798 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27638.37 MB 2025-02-14 09:46:44,798 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:46:44,798 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26450.52 MB 2025-02-14 09:46:44,805 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 09:46:44,805 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 09:46:44,805 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 09:46:44,805 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:46:44,805 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22509.61 MB 2025-02-14 09:46:44,805 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23024.38 MB 2025-02-14 09:46:44,805 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 514.77 MB 2025-02-14 09:46:44,805 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27638.37 MB 2025-02-14 09:46:44,805 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27638.37 MB 2025-02-14 09:46:44,805 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:46:44,805 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23410.64 MB 2025-02-14 09:46:44,916 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 09:46:44,916 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 09:46:44,916 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 09:46:44,916 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:46:44,916 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23024.38 MB 2025-02-14 09:46:44,916 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23650.52 MB 2025-02-14 09:46:44,916 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 626.14 MB 2025-02-14 09:46:44,916 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27638.37 MB 2025-02-14 09:46:44,916 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27638.37 MB 2025-02-14 09:46:44,916 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:46:44,916 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25146.10 MB 2025-02-14 09:46:44,917 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 09:46:44,917 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 09:46:44,917 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 09:46:44,917 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:46:44,917 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22509.61 MB 2025-02-14 09:46:44,917 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23650.52 MB 2025-02-14 09:46:44,917 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1140.92 MB 2025-02-14 09:46:44,917 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27638.37 MB 2025-02-14 09:46:44,917 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27638.37 MB 2025-02-14 09:46:44,917 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:46:44,917 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25146.10 MB 2025-02-14 09:46:44,976 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 09:46:44,976 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 09:46:44,976 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 09:46:44,976 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:46:44,976 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24254.67 MB 2025-02-14 09:46:44,976 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24517.35 MB 2025-02-14 09:46:44,976 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 262.68 MB 2025-02-14 09:46:44,976 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27638.37 MB 2025-02-14 09:46:44,976 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27804.04 MB 2025-02-14 09:46:44,976 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 165.68 MB 2025-02-14 09:46:44,976 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24710.22 MB 2025-02-14 09:46:44,982 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 09:46:44,982 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 09:46:44,982 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 09:46:44,982 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:46:44,982 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24683.45 MB 2025-02-14 09:46:44,982 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24911.99 MB 2025-02-14 09:46:44,982 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.54 MB 2025-02-14 09:46:44,982 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27804.04 MB 2025-02-14 09:46:44,982 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27804.04 MB 2025-02-14 09:46:44,982 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:46:44,982 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24911.99 MB 2025-02-14 09:46:44,983 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 09:46:44,984 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 09:46:44,984 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.40 seconds 2025-02-14 09:46:44,984 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:46:44,984 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21412.62 MB 2025-02-14 09:46:44,984 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25112.69 MB 2025-02-14 09:46:44,984 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3700.08 MB 2025-02-14 09:46:44,984 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62253.96 MB 2025-02-14 09:46:44,984 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27804.04 MB 2025-02-14 09:46:44,984 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34449.92 MB 2025-02-14 09:46:44,984 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25112.69 MB 2025-02-14 09:46:45,257 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 09:46:45,257 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 09:46:45,257 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 09:46:45,257 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:46:45,257 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25112.69 MB 2025-02-14 09:46:45,257 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28121.20 MB 2025-02-14 09:46:45,257 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3008.50 MB 2025-02-14 09:46:45,257 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27804.04 MB 2025-02-14 09:46:45,257 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29548.87 MB 2025-02-14 09:46:45,257 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1744.83 MB 2025-02-14 09:46:45,257 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28422.52 MB 2025-02-14 09:46:45,275 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8147, cut from 8149 2025-02-14 09:46:45,275 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2.'] 2025-02-14 09:46:45,281 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 09:46:45,281 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 09:46:45,281 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:46:45,281 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:46:45,281 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28121.20 MB 2025-02-14 09:46:45,281 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36544.40 MB 2025-02-14 09:46:45,281 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8423.21 MB 2025-02-14 09:46:45,282 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29548.87 MB 2025-02-14 09:46:45,282 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40019.95 MB 2025-02-14 09:46:45,282 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10471.08 MB 2025-02-14 09:46:45,282 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36544.40 MB 2025-02-14 09:46:45,444 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7939] 2025-02-14 09:46:45,445 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:46:45,445 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 09:46:45,446 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:46:45,446 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 09:46:45,451 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 09:46:45,452 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:46:45,452 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 09:46:45,452 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2.'] 2025-02-14 09:47:50,464 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:47:50,464 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 09:47:50,469 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 09:47:50,473 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:47:50,473 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1434, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 09:47:50,474 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:47:50,474 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1434, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 09:48:12,394 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 09:48:12,394 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 09:48:12,394 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.91 seconds 2025-02-14 09:48:12,394 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:48:12,394 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31025.19 MB 2025-02-14 09:48:12,394 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36100.30 MB 2025-02-14 09:48:12,394 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5075.11 MB 2025-02-14 09:48:12,394 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48398.07 MB 2025-02-14 09:48:12,394 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48526.00 MB 2025-02-14 09:48:12,394 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 127.93 MB 2025-02-14 09:48:12,394 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45026.41 MB 2025-02-14 09:48:12,476 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 09:48:12,476 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 09:48:12,476 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 09:48:12,476 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:48:12,476 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36100.30 MB 2025-02-14 09:48:12,476 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31296.91 MB 2025-02-14 09:48:12,476 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4803.39 MB 2025-02-14 09:48:12,476 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48526.00 MB 2025-02-14 09:48:12,476 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58487.47 MB 2025-02-14 09:48:12,476 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9961.47 MB 2025-02-14 09:48:12,476 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51113.52 MB 2025-02-14 09:48:14,390 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 09:48:14,390 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 09:48:14,390 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 09:48:14,390 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:48:14,390 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31296.91 MB 2025-02-14 09:48:14,390 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31827.76 MB 2025-02-14 09:48:14,390 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 09:48:14,390 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58487.47 MB 2025-02-14 09:48:14,390 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43450.89 MB 2025-02-14 09:48:14,390 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15036.58 MB 2025-02-14 09:48:14,390 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35807.09 MB 2025-02-14 09:48:14,403 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 09:48:14,403 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 09:48:14,403 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:48:14,403 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:48:14,403 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31827.76 MB 2025-02-14 09:48:14,403 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33717.11 MB 2025-02-14 09:48:14,404 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.35 MB 2025-02-14 09:48:14,404 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43450.89 MB 2025-02-14 09:48:14,404 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43450.89 MB 2025-02-14 09:48:14,404 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:48:14,404 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35134.54 MB 2025-02-14 09:48:14,613 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 09:48:14,613 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 09:48:14,613 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 09:48:14,613 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:48:14,613 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33717.11 MB 2025-02-14 09:48:14,613 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35958.97 MB 2025-02-14 09:48:14,613 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 09:48:14,613 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43450.89 MB 2025-02-14 09:48:14,613 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47225.77 MB 2025-02-14 09:48:14,613 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 09:48:14,613 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41504.15 MB 2025-02-14 09:48:14,614 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 09:48:14,614 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 09:48:14,614 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 09:48:14,614 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:48:14,614 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31827.76 MB 2025-02-14 09:48:14,614 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35958.97 MB 2025-02-14 09:48:14,614 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.21 MB 2025-02-14 09:48:14,614 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43450.89 MB 2025-02-14 09:48:14,614 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47225.77 MB 2025-02-14 09:48:14,614 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 09:48:14,614 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41504.15 MB 2025-02-14 09:48:14,780 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 09:48:14,780 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 09:48:14,780 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 09:48:14,780 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:48:14,780 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37493.41 MB 2025-02-14 09:48:14,780 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38260.41 MB 2025-02-14 09:48:14,780 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 09:48:14,780 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47225.77 MB 2025-02-14 09:48:14,780 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47636.81 MB 2025-02-14 09:48:14,780 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 411.04 MB 2025-02-14 09:48:14,780 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38968.20 MB 2025-02-14 09:48:14,799 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 09:48:14,799 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 09:48:14,799 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:48:14,799 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:48:14,799 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38673.30 MB 2025-02-14 09:48:14,799 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38901.42 MB 2025-02-14 09:48:14,799 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.13 MB 2025-02-14 09:48:14,799 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47636.81 MB 2025-02-14 09:48:14,799 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47636.81 MB 2025-02-14 09:48:14,799 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:48:14,799 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39143.29 MB 2025-02-14 09:48:14,800 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 09:48:14,800 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 09:48:14,800 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.32 seconds 2025-02-14 09:48:14,800 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:48:14,800 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26029.02 MB 2025-02-14 09:48:14,800 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39101.46 MB 2025-02-14 09:48:14,800 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13072.44 MB 2025-02-14 09:48:14,800 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48398.07 MB 2025-02-14 09:48:14,800 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47636.81 MB 2025-02-14 09:48:14,800 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -761.27 MB 2025-02-14 09:48:14,800 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39143.29 MB 2025-02-14 09:48:15,067 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 09:48:15,067 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 09:48:15,067 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 09:48:15,067 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:48:15,067 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39101.46 MB 2025-02-14 09:48:15,067 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31018.83 MB 2025-02-14 09:48:15,067 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8082.64 MB 2025-02-14 09:48:15,067 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47636.81 MB 2025-02-14 09:48:15,067 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47636.81 MB 2025-02-14 09:48:15,067 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:48:15,067 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41600.23 MB 2025-02-14 09:48:15,088 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8120, cut from 8122 2025-02-14 09:48:15,088 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 09:48:15,095 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 09:48:15,095 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 09:48:15,095 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 09:48:15,095 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:48:15,095 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31018.83 MB 2025-02-14 09:48:15,095 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39415.47 MB 2025-02-14 09:48:15,095 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8396.64 MB 2025-02-14 09:48:15,095 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47636.81 MB 2025-02-14 09:48:15,095 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55983.47 MB 2025-02-14 09:48:15,095 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8346.66 MB 2025-02-14 09:48:15,095 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39415.47 MB 2025-02-14 09:48:15,260 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7912] 2025-02-14 09:48:15,262 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:48:15,262 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 09:48:15,263 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:48:15,263 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 09:48:15,268 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 09:48:15,269 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:48:15,269 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 09:48:15,269 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 09:48:23,522 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:48:23,522 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 09:48:23,527 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 09:48:23,530 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:48:23,530 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1605, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 09:48:23,531 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:48:23,531 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1605, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 09:48:48,399 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 09:48:48,399 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 09:48:48,399 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.86 seconds 2025-02-14 09:48:48,399 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:48:48,399 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32216.75 MB 2025-02-14 09:48:48,399 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37896.75 MB 2025-02-14 09:48:48,399 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5680.01 MB 2025-02-14 09:48:48,399 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64330.14 MB 2025-02-14 09:48:48,399 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49085.94 MB 2025-02-14 09:48:48,399 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15244.20 MB 2025-02-14 09:48:48,399 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46898.25 MB 2025-02-14 09:48:48,486 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 09:48:48,486 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 09:48:48,486 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 09:48:48,486 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:48:48,486 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37896.75 MB 2025-02-14 09:48:48,486 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32185.89 MB 2025-02-14 09:48:48,486 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5710.86 MB 2025-02-14 09:48:48,486 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49085.94 MB 2025-02-14 09:48:48,486 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 59762.54 MB 2025-02-14 09:48:48,486 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10676.60 MB 2025-02-14 09:48:48,486 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 53970.30 MB 2025-02-14 09:48:50,407 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 09:48:50,407 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 09:48:50,407 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 09:48:50,407 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:48:50,407 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32185.89 MB 2025-02-14 09:48:50,407 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32716.73 MB 2025-02-14 09:48:50,407 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 09:48:50,407 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59762.54 MB 2025-02-14 09:48:50,407 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43404.75 MB 2025-02-14 09:48:50,407 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16357.79 MB 2025-02-14 09:48:50,407 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36696.07 MB 2025-02-14 09:48:50,420 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 09:48:50,420 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 09:48:50,420 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:48:50,420 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:48:50,420 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32716.73 MB 2025-02-14 09:48:50,421 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34606.09 MB 2025-02-14 09:48:50,421 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.35 MB 2025-02-14 09:48:50,421 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43404.75 MB 2025-02-14 09:48:50,421 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43404.75 MB 2025-02-14 09:48:50,421 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:48:50,421 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36023.52 MB 2025-02-14 09:48:50,630 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 09:48:50,630 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 09:48:50,630 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 09:48:50,630 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:48:50,630 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34606.09 MB 2025-02-14 09:48:50,630 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36847.94 MB 2025-02-14 09:48:50,630 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 09:48:50,630 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43404.75 MB 2025-02-14 09:48:50,630 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46235.91 MB 2025-02-14 09:48:50,630 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-14 09:48:50,630 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42393.12 MB 2025-02-14 09:48:50,631 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 09:48:50,631 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 09:48:50,631 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 09:48:50,631 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:48:50,631 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32716.73 MB 2025-02-14 09:48:50,631 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36847.94 MB 2025-02-14 09:48:50,631 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.21 MB 2025-02-14 09:48:50,631 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43404.75 MB 2025-02-14 09:48:50,631 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46235.91 MB 2025-02-14 09:48:50,631 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-14 09:48:50,631 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42393.12 MB 2025-02-14 09:48:50,797 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 09:48:50,797 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 09:48:50,797 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 09:48:50,797 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:48:50,797 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38382.38 MB 2025-02-14 09:48:50,797 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39149.39 MB 2025-02-14 09:48:50,797 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 09:48:50,797 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46235.91 MB 2025-02-14 09:48:50,797 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46646.95 MB 2025-02-14 09:48:50,797 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 411.04 MB 2025-02-14 09:48:50,797 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39857.18 MB 2025-02-14 09:48:50,816 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 09:48:50,816 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 09:48:50,816 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:48:50,816 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:48:50,816 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39562.28 MB 2025-02-14 09:48:50,816 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39789.52 MB 2025-02-14 09:48:50,816 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.24 MB 2025-02-14 09:48:50,816 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46646.95 MB 2025-02-14 09:48:50,816 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46646.95 MB 2025-02-14 09:48:50,816 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:48:50,816 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40025.03 MB 2025-02-14 09:48:50,817 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 09:48:50,817 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 09:48:50,817 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.28 seconds 2025-02-14 09:48:50,817 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:48:50,817 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26624.80 MB 2025-02-14 09:48:50,817 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39989.73 MB 2025-02-14 09:48:50,817 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13364.93 MB 2025-02-14 09:48:50,817 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64330.14 MB 2025-02-14 09:48:50,817 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46646.95 MB 2025-02-14 09:48:50,817 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17683.19 MB 2025-02-14 09:48:50,817 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40025.03 MB 2025-02-14 09:48:51,086 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 09:48:51,086 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 09:48:51,086 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 09:48:51,086 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:48:51,086 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39989.73 MB 2025-02-14 09:48:51,086 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31617.19 MB 2025-02-14 09:48:51,086 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8372.54 MB 2025-02-14 09:48:51,086 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46646.95 MB 2025-02-14 09:48:51,086 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46646.95 MB 2025-02-14 09:48:51,087 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:48:51,087 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42490.64 MB 2025-02-14 09:48:51,111 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8127, cut from 8129 2025-02-14 09:48:51,111 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 09:48:51,118 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 09:48:51,118 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 09:48:51,118 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 09:48:51,118 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:48:51,118 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31617.19 MB 2025-02-14 09:48:51,118 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40021.13 MB 2025-02-14 09:48:51,119 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8403.94 MB 2025-02-14 09:48:51,119 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46646.95 MB 2025-02-14 09:48:51,119 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55002.01 MB 2025-02-14 09:48:51,119 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8355.05 MB 2025-02-14 09:48:51,119 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40021.13 MB 2025-02-14 09:48:51,283 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7919] 2025-02-14 09:48:51,285 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:48:51,285 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 09:48:51,286 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:48:51,286 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 09:48:51,290 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 09:48:51,291 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:48:51,292 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 09:48:51,292 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 09:49:18,782 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:49:18,782 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 09:49:18,789 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 09:49:18,796 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:49:18,796 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 147, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 09:49:18,798 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:49:18,798 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 147, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 09:49:21,164 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 09:49:21,164 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 09:49:21,164 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.36 seconds 2025-02-14 09:49:21,164 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:49:21,164 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22057.17 MB 2025-02-14 09:49:21,164 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22577.40 MB 2025-02-14 09:49:21,164 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 520.22 MB 2025-02-14 09:49:21,164 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63357.06 MB 2025-02-14 09:49:21,164 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30867.98 MB 2025-02-14 09:49:21,164 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32489.08 MB 2025-02-14 09:49:21,164 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31528.54 MB 2025-02-14 09:49:21,179 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 09:49:21,180 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 09:49:21,180 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:49:21,180 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:49:21,180 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22577.40 MB 2025-02-14 09:49:21,180 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22815.40 MB 2025-02-14 09:49:21,180 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 238.00 MB 2025-02-14 09:49:21,180 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30867.98 MB 2025-02-14 09:49:21,180 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30867.98 MB 2025-02-14 09:49:21,180 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:49:21,180 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24624.76 MB 2025-02-14 09:49:21,907 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 09:49:21,907 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 09:49:21,907 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.72 seconds 2025-02-14 09:49:21,907 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:49:21,907 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22815.40 MB 2025-02-14 09:49:21,907 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23007.83 MB 2025-02-14 09:49:21,907 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 192.43 MB 2025-02-14 09:49:21,907 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30867.98 MB 2025-02-14 09:49:21,907 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30867.98 MB 2025-02-14 09:49:21,907 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:49:21,907 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26985.84 MB 2025-02-14 09:49:21,917 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 09:49:21,917 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 09:49:21,917 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 09:49:21,918 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:49:21,918 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23007.76 MB 2025-02-14 09:49:21,918 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23692.55 MB 2025-02-14 09:49:21,918 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 684.79 MB 2025-02-14 09:49:21,918 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30867.98 MB 2025-02-14 09:49:21,918 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30867.98 MB 2025-02-14 09:49:21,918 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:49:21,918 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24206.38 MB 2025-02-14 09:49:22,024 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 09:49:22,024 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 09:49:22,024 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 09:49:22,024 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:49:22,024 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23692.55 MB 2025-02-14 09:49:22,024 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24505.27 MB 2025-02-14 09:49:22,024 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 812.71 MB 2025-02-14 09:49:22,024 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30867.98 MB 2025-02-14 09:49:22,024 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30867.98 MB 2025-02-14 09:49:22,024 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:49:22,024 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26515.03 MB 2025-02-14 09:49:22,025 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 09:49:22,025 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 09:49:22,025 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 09:49:22,025 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:49:22,025 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23007.76 MB 2025-02-14 09:49:22,025 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24505.27 MB 2025-02-14 09:49:22,025 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1497.50 MB 2025-02-14 09:49:22,025 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30867.98 MB 2025-02-14 09:49:22,025 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30867.98 MB 2025-02-14 09:49:22,026 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:49:22,026 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26515.03 MB 2025-02-14 09:49:22,130 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 09:49:22,130 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 09:49:22,130 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 09:49:22,130 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:49:22,130 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25062.02 MB 2025-02-14 09:49:22,130 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25340.06 MB 2025-02-14 09:49:22,130 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 278.04 MB 2025-02-14 09:49:22,130 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30867.98 MB 2025-02-14 09:49:22,130 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31014.78 MB 2025-02-14 09:49:22,130 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 146.80 MB 2025-02-14 09:49:22,130 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25607.71 MB 2025-02-14 09:49:22,145 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 09:49:22,145 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 09:49:22,145 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:49:22,145 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:49:22,145 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25489.74 MB 2025-02-14 09:49:22,145 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25719.03 MB 2025-02-14 09:49:22,145 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.29 MB 2025-02-14 09:49:22,145 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31014.78 MB 2025-02-14 09:49:22,145 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31014.78 MB 2025-02-14 09:49:22,145 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:49:22,145 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25734.25 MB 2025-02-14 09:49:22,147 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 09:49:22,147 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 09:49:22,147 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.35 seconds 2025-02-14 09:49:22,147 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:49:22,147 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21545.01 MB 2025-02-14 09:49:22,147 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25920.10 MB 2025-02-14 09:49:22,147 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4375.09 MB 2025-02-14 09:49:22,147 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63357.06 MB 2025-02-14 09:49:22,147 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31014.78 MB 2025-02-14 09:49:22,147 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32342.28 MB 2025-02-14 09:49:22,147 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25920.10 MB 2025-02-14 09:49:22,435 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 09:49:22,435 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 09:49:22,435 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-14 09:49:22,435 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:49:22,435 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25920.10 MB 2025-02-14 09:49:22,435 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25346.64 MB 2025-02-14 09:49:22,435 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -573.46 MB 2025-02-14 09:49:22,435 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31014.78 MB 2025-02-14 09:49:22,435 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31014.78 MB 2025-02-14 09:49:22,435 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:49:22,435 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27025.23 MB 2025-02-14 09:49:22,455 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 09:49:22,455 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2,'] 2025-02-14 09:49:22,463 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 09:49:22,463 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 09:49:22,463 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:49:22,463 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:49:22,463 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25346.64 MB 2025-02-14 09:49:22,463 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33785.67 MB 2025-02-14 09:49:22,463 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 09:49:22,463 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31014.78 MB 2025-02-14 09:49:22,463 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39405.49 MB 2025-02-14 09:49:22,463 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 09:49:22,463 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33785.67 MB 2025-02-14 09:49:22,724 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 09:49:22,727 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:49:22,727 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 09:49:22,728 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:49:22,729 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 09:49:22,736 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 09:49:22,738 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:49:22,738 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 09:49:22,739 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2,'] 2025-02-14 09:49:55,593 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:49:55,593 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 09:49:55,600 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 09:49:55,605 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:49:55,606 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 570, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 09:49:55,607 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:49:55,607 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 570, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 09:50:04,419 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 09:50:04,419 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 09:50:04,419 - resource_logging.py:150 - __exit__ - DEBUG - Time: 8.81 seconds 2025-02-14 09:50:04,419 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:50:04,419 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25004.70 MB 2025-02-14 09:50:04,419 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27022.16 MB 2025-02-14 09:50:04,419 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2017.46 MB 2025-02-14 09:50:04,419 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51990.50 MB 2025-02-14 09:50:04,419 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32885.44 MB 2025-02-14 09:50:04,419 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19105.05 MB 2025-02-14 09:50:04,419 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35835.83 MB 2025-02-14 09:50:04,457 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 09:50:04,457 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 09:50:04,457 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-14 09:50:04,457 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:50:04,457 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27022.16 MB 2025-02-14 09:50:04,457 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26806.30 MB 2025-02-14 09:50:04,457 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -215.87 MB 2025-02-14 09:50:04,457 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32885.44 MB 2025-02-14 09:50:04,457 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39367.74 MB 2025-02-14 09:50:04,457 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6482.30 MB 2025-02-14 09:50:04,457 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35216.60 MB 2025-02-14 09:50:06,360 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 09:50:06,360 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 09:50:06,360 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.90 seconds 2025-02-14 09:50:06,360 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:50:06,360 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26806.30 MB 2025-02-14 09:50:06,360 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27337.14 MB 2025-02-14 09:50:06,360 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 09:50:06,360 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39367.74 MB 2025-02-14 09:50:06,360 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32992.40 MB 2025-02-14 09:50:06,360 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6375.34 MB 2025-02-14 09:50:06,360 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31316.47 MB 2025-02-14 09:50:06,374 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 09:50:06,374 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 09:50:06,374 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:50:06,374 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:50:06,374 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27337.14 MB 2025-02-14 09:50:06,374 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29226.49 MB 2025-02-14 09:50:06,374 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.35 MB 2025-02-14 09:50:06,374 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32992.40 MB 2025-02-14 09:50:06,374 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33936.11 MB 2025-02-14 09:50:06,374 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 09:50:06,374 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30643.92 MB 2025-02-14 09:50:06,584 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 09:50:06,584 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 09:50:06,584 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 09:50:06,584 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:50:06,584 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29226.49 MB 2025-02-14 09:50:06,584 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31468.35 MB 2025-02-14 09:50:06,584 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 09:50:06,584 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33936.11 MB 2025-02-14 09:50:06,584 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39598.42 MB 2025-02-14 09:50:06,584 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 09:50:06,584 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37013.53 MB 2025-02-14 09:50:06,585 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 09:50:06,585 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 09:50:06,585 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 09:50:06,585 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:50:06,585 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27337.14 MB 2025-02-14 09:50:06,585 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31468.35 MB 2025-02-14 09:50:06,585 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.21 MB 2025-02-14 09:50:06,585 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32992.40 MB 2025-02-14 09:50:06,585 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39598.42 MB 2025-02-14 09:50:06,585 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 09:50:06,585 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37013.53 MB 2025-02-14 09:50:06,754 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 09:50:06,754 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 09:50:06,754 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 09:50:06,754 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:50:06,754 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33002.79 MB 2025-02-14 09:50:06,754 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33769.79 MB 2025-02-14 09:50:06,754 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 09:50:06,754 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39598.42 MB 2025-02-14 09:50:06,754 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40011.56 MB 2025-02-14 09:50:06,754 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 09:50:06,754 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34477.58 MB 2025-02-14 09:50:06,772 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 09:50:06,773 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 09:50:06,773 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:50:06,773 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:50:06,773 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34182.68 MB 2025-02-14 09:50:06,773 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34412.18 MB 2025-02-14 09:50:06,773 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.50 MB 2025-02-14 09:50:06,773 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40011.56 MB 2025-02-14 09:50:06,773 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40011.56 MB 2025-02-14 09:50:06,773 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:50:06,773 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34624.16 MB 2025-02-14 09:50:06,774 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 09:50:06,774 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 09:50:06,774 - resource_logging.py:150 - __exit__ - DEBUG - Time: 11.16 seconds 2025-02-14 09:50:06,774 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:50:06,774 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23018.78 MB 2025-02-14 09:50:06,774 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34613.26 MB 2025-02-14 09:50:06,774 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11594.48 MB 2025-02-14 09:50:06,774 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51990.50 MB 2025-02-14 09:50:06,774 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40011.56 MB 2025-02-14 09:50:06,774 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11978.93 MB 2025-02-14 09:50:06,774 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34624.16 MB 2025-02-14 09:50:07,043 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 09:50:07,043 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 09:50:07,043 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 09:50:07,043 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:50:07,043 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34613.26 MB 2025-02-14 09:50:07,043 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28024.07 MB 2025-02-14 09:50:07,043 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6589.19 MB 2025-02-14 09:50:07,043 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40011.56 MB 2025-02-14 09:50:07,043 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40011.56 MB 2025-02-14 09:50:07,043 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:50:07,043 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37124.92 MB 2025-02-14 09:50:07,061 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 09:50:07,061 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 09:50:07,067 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 09:50:07,067 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 09:50:07,067 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:50:07,067 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:50:07,067 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28024.07 MB 2025-02-14 09:50:07,067 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36463.09 MB 2025-02-14 09:50:07,067 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 09:50:07,067 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40011.56 MB 2025-02-14 09:50:07,067 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50501.52 MB 2025-02-14 09:50:07,067 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 09:50:07,067 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36463.09 MB 2025-02-14 09:50:07,230 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 09:50:07,231 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:50:07,231 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 09:50:07,232 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:50:07,232 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 09:50:07,237 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 09:50:07,238 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:50:07,238 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 09:50:07,238 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 09:50:58,110 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:50:58,111 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 09:50:58,118 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 09:50:58,125 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:50:58,125 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 675, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 09:50:58,127 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:50:58,127 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 675, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 09:51:08,641 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 09:51:08,641 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 09:51:08,641 - resource_logging.py:150 - __exit__ - DEBUG - Time: 10.51 seconds 2025-02-14 09:51:08,641 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:51:08,641 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25736.36 MB 2025-02-14 09:51:08,641 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28125.15 MB 2025-02-14 09:51:08,641 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2388.79 MB 2025-02-14 09:51:08,641 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63086.53 MB 2025-02-14 09:51:08,641 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34676.41 MB 2025-02-14 09:51:08,641 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28410.12 MB 2025-02-14 09:51:08,641 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37020.48 MB 2025-02-14 09:51:08,682 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 09:51:08,682 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 09:51:08,682 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-14 09:51:08,682 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:51:08,682 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28125.15 MB 2025-02-14 09:51:08,682 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27351.11 MB 2025-02-14 09:51:08,682 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -774.04 MB 2025-02-14 09:51:08,682 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34676.41 MB 2025-02-14 09:51:08,682 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40472.94 MB 2025-02-14 09:51:08,682 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5796.53 MB 2025-02-14 09:51:08,682 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36322.06 MB 2025-02-14 09:51:10,589 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 09:51:10,589 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 09:51:10,589 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 09:51:10,589 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:51:10,589 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27351.11 MB 2025-02-14 09:51:10,589 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27881.95 MB 2025-02-14 09:51:10,589 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 09:51:10,589 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40472.94 MB 2025-02-14 09:51:10,589 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33701.23 MB 2025-02-14 09:51:10,589 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6771.70 MB 2025-02-14 09:51:10,589 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31861.29 MB 2025-02-14 09:51:10,603 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 09:51:10,603 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 09:51:10,603 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:51:10,603 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:51:10,603 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27881.95 MB 2025-02-14 09:51:10,603 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29771.31 MB 2025-02-14 09:51:10,603 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.35 MB 2025-02-14 09:51:10,603 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33701.23 MB 2025-02-14 09:51:10,603 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34644.95 MB 2025-02-14 09:51:10,603 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 09:51:10,603 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31188.73 MB 2025-02-14 09:51:10,817 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 09:51:10,817 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 09:51:10,817 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 09:51:10,817 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:51:10,817 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29771.31 MB 2025-02-14 09:51:10,817 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32013.16 MB 2025-02-14 09:51:10,817 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 09:51:10,817 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34644.95 MB 2025-02-14 09:51:10,817 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40307.26 MB 2025-02-14 09:51:10,817 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 09:51:10,817 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37558.34 MB 2025-02-14 09:51:10,817 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 09:51:10,817 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 09:51:10,817 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 09:51:10,817 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:51:10,817 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27881.95 MB 2025-02-14 09:51:10,817 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32013.16 MB 2025-02-14 09:51:10,817 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.21 MB 2025-02-14 09:51:10,817 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33701.23 MB 2025-02-14 09:51:10,818 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40307.26 MB 2025-02-14 09:51:10,818 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 09:51:10,818 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37558.34 MB 2025-02-14 09:51:11,031 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 09:51:11,031 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 09:51:11,031 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 09:51:11,031 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:51:11,031 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33547.60 MB 2025-02-14 09:51:11,031 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34314.61 MB 2025-02-14 09:51:11,031 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 09:51:11,031 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40307.26 MB 2025-02-14 09:51:11,031 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40718.30 MB 2025-02-14 09:51:11,031 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 411.04 MB 2025-02-14 09:51:11,031 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35022.39 MB 2025-02-14 09:51:11,057 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 09:51:11,057 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 09:51:11,057 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:51:11,057 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:51:11,057 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34727.50 MB 2025-02-14 09:51:11,057 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34958.62 MB 2025-02-14 09:51:11,057 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 231.12 MB 2025-02-14 09:51:11,057 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40718.30 MB 2025-02-14 09:51:11,057 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40718.30 MB 2025-02-14 09:51:11,057 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:51:11,057 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35149.68 MB 2025-02-14 09:51:11,059 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 09:51:11,059 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 09:51:11,059 - resource_logging.py:150 - __exit__ - DEBUG - Time: 12.93 seconds 2025-02-14 09:51:11,059 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:51:11,059 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23384.61 MB 2025-02-14 09:51:11,059 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35159.69 MB 2025-02-14 09:51:11,059 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11775.09 MB 2025-02-14 09:51:11,059 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63086.53 MB 2025-02-14 09:51:11,059 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40718.30 MB 2025-02-14 09:51:11,059 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22368.22 MB 2025-02-14 09:51:11,059 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35159.69 MB 2025-02-14 09:51:11,341 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 09:51:11,341 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 09:51:11,341 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.28 seconds 2025-02-14 09:51:11,341 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:51:11,341 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35159.69 MB 2025-02-14 09:51:11,341 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28389.90 MB 2025-02-14 09:51:11,341 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6769.80 MB 2025-02-14 09:51:11,341 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40718.30 MB 2025-02-14 09:51:11,341 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40718.30 MB 2025-02-14 09:51:11,341 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:51:11,341 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37671.36 MB 2025-02-14 09:51:11,360 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 09:51:11,361 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 09:51:11,376 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 09:51:11,376 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 09:51:11,376 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 09:51:11,376 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:51:11,377 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28389.90 MB 2025-02-14 09:51:11,377 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36828.92 MB 2025-02-14 09:51:11,377 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 09:51:11,377 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40718.30 MB 2025-02-14 09:51:11,377 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51208.26 MB 2025-02-14 09:51:11,377 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 09:51:11,377 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36828.92 MB 2025-02-14 09:51:11,601 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 09:51:11,604 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:51:11,604 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 09:51:11,606 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:51:11,606 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 09:51:11,612 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 09:51:11,614 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:51:11,614 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 09:51:11,614 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 09:51:19,836 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:51:19,836 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 09:51:19,841 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 09:51:19,844 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:51:19,844 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1297, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 09:51:19,845 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:51:19,845 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1297, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 09:51:39,871 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 09:51:39,871 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 09:51:39,871 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.02 seconds 2025-02-14 09:51:39,871 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:51:39,871 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30070.56 MB 2025-02-14 09:51:39,871 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34661.22 MB 2025-02-14 09:51:39,871 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4590.67 MB 2025-02-14 09:51:39,871 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63793.27 MB 2025-02-14 09:51:39,871 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48043.66 MB 2025-02-14 09:51:39,871 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15749.61 MB 2025-02-14 09:51:39,871 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43618.79 MB 2025-02-14 09:51:39,936 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 09:51:39,936 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 09:51:39,936 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 09:51:39,936 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:51:39,936 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34661.22 MB 2025-02-14 09:51:39,936 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30584.69 MB 2025-02-14 09:51:39,936 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4076.53 MB 2025-02-14 09:51:39,936 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48043.66 MB 2025-02-14 09:51:39,936 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51032.10 MB 2025-02-14 09:51:39,936 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2988.44 MB 2025-02-14 09:51:39,936 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44816.98 MB 2025-02-14 09:51:41,859 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 09:51:41,859 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 09:51:41,859 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 09:51:41,859 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:51:41,859 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30584.69 MB 2025-02-14 09:51:41,859 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31115.54 MB 2025-02-14 09:51:41,859 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 09:51:41,859 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51032.10 MB 2025-02-14 09:51:41,859 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39279.66 MB 2025-02-14 09:51:41,859 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11752.44 MB 2025-02-14 09:51:41,859 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35094.87 MB 2025-02-14 09:51:41,873 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 09:51:41,873 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 09:51:41,873 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:51:41,873 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:51:41,873 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31115.54 MB 2025-02-14 09:51:41,873 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33004.89 MB 2025-02-14 09:51:41,873 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.35 MB 2025-02-14 09:51:41,873 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39279.66 MB 2025-02-14 09:51:41,873 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39279.66 MB 2025-02-14 09:51:41,873 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:51:41,873 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34422.32 MB 2025-02-14 09:51:42,086 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 09:51:42,086 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 09:51:42,086 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 09:51:42,086 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:51:42,086 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33004.89 MB 2025-02-14 09:51:42,086 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35246.75 MB 2025-02-14 09:51:42,086 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 09:51:42,086 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39279.66 MB 2025-02-14 09:51:42,086 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43998.25 MB 2025-02-14 09:51:42,086 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 09:51:42,086 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40791.93 MB 2025-02-14 09:51:42,087 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 09:51:42,087 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 09:51:42,087 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 09:51:42,087 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:51:42,087 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31115.54 MB 2025-02-14 09:51:42,087 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35246.75 MB 2025-02-14 09:51:42,087 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.21 MB 2025-02-14 09:51:42,087 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39279.66 MB 2025-02-14 09:51:42,087 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43998.25 MB 2025-02-14 09:51:42,087 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 09:51:42,087 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40791.93 MB 2025-02-14 09:51:42,254 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 09:51:42,254 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 09:51:42,254 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 09:51:42,254 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:51:42,254 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36781.19 MB 2025-02-14 09:51:42,254 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37548.19 MB 2025-02-14 09:51:42,254 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 09:51:42,254 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43998.25 MB 2025-02-14 09:51:42,254 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44409.29 MB 2025-02-14 09:51:42,254 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 411.04 MB 2025-02-14 09:51:42,254 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38255.98 MB 2025-02-14 09:51:42,274 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 09:51:42,274 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 09:51:42,274 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:51:42,274 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:51:42,274 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37961.08 MB 2025-02-14 09:51:42,274 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38190.93 MB 2025-02-14 09:51:42,274 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.85 MB 2025-02-14 09:51:42,274 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44409.29 MB 2025-02-14 09:51:42,274 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44409.29 MB 2025-02-14 09:51:42,274 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:51:42,274 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38414.42 MB 2025-02-14 09:51:42,275 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 09:51:42,275 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 09:51:42,275 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.43 seconds 2025-02-14 09:51:42,275 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:51:42,275 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25551.70 MB 2025-02-14 09:51:42,275 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38392.00 MB 2025-02-14 09:51:42,275 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12840.29 MB 2025-02-14 09:51:42,275 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63793.27 MB 2025-02-14 09:51:42,275 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44409.29 MB 2025-02-14 09:51:42,275 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19383.98 MB 2025-02-14 09:51:42,275 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38414.42 MB 2025-02-14 09:51:42,544 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 09:51:42,544 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 09:51:42,544 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 09:51:42,544 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:51:42,544 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38392.00 MB 2025-02-14 09:51:42,544 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30556.99 MB 2025-02-14 09:51:42,544 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7835.00 MB 2025-02-14 09:51:42,544 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44409.29 MB 2025-02-14 09:51:42,544 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44409.29 MB 2025-02-14 09:51:42,544 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:51:42,544 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40903.67 MB 2025-02-14 09:51:42,562 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 09:51:42,562 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 09:51:42,568 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 09:51:42,568 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 09:51:42,568 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:51:42,568 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:51:42,568 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30556.99 MB 2025-02-14 09:51:42,568 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38996.02 MB 2025-02-14 09:51:42,568 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 09:51:42,568 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44409.29 MB 2025-02-14 09:51:42,568 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48605.69 MB 2025-02-14 09:51:42,568 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4196.40 MB 2025-02-14 09:51:42,568 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38996.02 MB 2025-02-14 09:51:42,731 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 09:51:42,732 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:51:42,732 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 09:51:42,733 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:51:42,733 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 09:51:42,738 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 09:51:42,739 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:51:42,739 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 09:51:42,739 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 09:53:56,364 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:53:56,365 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 09:53:56,370 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 09:53:56,375 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:53:56,375 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 158, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 09:53:56,376 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:53:56,376 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 158, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 09:53:58,827 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 09:53:58,827 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 09:53:58,827 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.45 seconds 2025-02-14 09:53:58,827 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:53:58,827 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22133.82 MB 2025-02-14 09:53:58,827 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22692.97 MB 2025-02-14 09:53:58,827 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 559.15 MB 2025-02-14 09:53:58,827 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61190.70 MB 2025-02-14 09:53:58,827 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26694.65 MB 2025-02-14 09:53:58,827 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34496.05 MB 2025-02-14 09:53:58,827 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31606.00 MB 2025-02-14 09:53:58,838 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 09:53:58,838 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 09:53:58,838 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:53:58,838 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:53:58,838 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22692.97 MB 2025-02-14 09:53:58,838 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22872.58 MB 2025-02-14 09:53:58,838 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 179.61 MB 2025-02-14 09:53:58,838 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26694.65 MB 2025-02-14 09:53:58,838 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26694.65 MB 2025-02-14 09:53:58,838 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:53:58,838 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24779.26 MB 2025-02-14 09:53:59,536 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 09:53:59,536 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 09:53:59,536 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.70 seconds 2025-02-14 09:53:59,536 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:53:59,536 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22872.58 MB 2025-02-14 09:53:59,536 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23065.01 MB 2025-02-14 09:53:59,536 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 192.43 MB 2025-02-14 09:53:59,536 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26694.65 MB 2025-02-14 09:53:59,536 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27097.30 MB 2025-02-14 09:53:59,536 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 402.65 MB 2025-02-14 09:53:59,536 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27043.02 MB 2025-02-14 09:53:59,543 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 09:53:59,543 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 09:53:59,543 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 09:53:59,543 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:53:59,543 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23064.95 MB 2025-02-14 09:53:59,543 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23749.74 MB 2025-02-14 09:53:59,543 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 684.79 MB 2025-02-14 09:53:59,543 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27097.30 MB 2025-02-14 09:53:59,543 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27097.30 MB 2025-02-14 09:53:59,543 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:53:59,543 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24263.56 MB 2025-02-14 09:53:59,620 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 09:53:59,620 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 09:53:59,620 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 09:53:59,621 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:53:59,621 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23749.74 MB 2025-02-14 09:53:59,621 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24562.45 MB 2025-02-14 09:53:59,621 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 812.71 MB 2025-02-14 09:53:59,621 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27097.30 MB 2025-02-14 09:53:59,621 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28301.07 MB 2025-02-14 09:53:59,621 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1203.77 MB 2025-02-14 09:53:59,621 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26573.13 MB 2025-02-14 09:53:59,621 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 09:53:59,621 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 09:53:59,621 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 09:53:59,621 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:53:59,621 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23064.95 MB 2025-02-14 09:53:59,621 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24562.45 MB 2025-02-14 09:53:59,621 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1497.50 MB 2025-02-14 09:53:59,621 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27097.30 MB 2025-02-14 09:53:59,621 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28301.07 MB 2025-02-14 09:53:59,621 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1203.77 MB 2025-02-14 09:53:59,621 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26573.13 MB 2025-02-14 09:53:59,680 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 09:53:59,680 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 09:53:59,680 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 09:53:59,680 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:53:59,680 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25118.36 MB 2025-02-14 09:53:59,680 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25397.32 MB 2025-02-14 09:53:59,680 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 278.96 MB 2025-02-14 09:53:59,681 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28301.07 MB 2025-02-14 09:53:59,681 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28447.87 MB 2025-02-14 09:53:59,681 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 146.80 MB 2025-02-14 09:53:59,681 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25663.63 MB 2025-02-14 09:53:59,689 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 09:53:59,689 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 09:53:59,689 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:53:59,689 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:53:59,689 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25547.00 MB 2025-02-14 09:53:59,689 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25774.93 MB 2025-02-14 09:53:59,689 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.94 MB 2025-02-14 09:53:59,689 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28447.87 MB 2025-02-14 09:53:59,689 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28447.87 MB 2025-02-14 09:53:59,689 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:53:59,689 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25784.13 MB 2025-02-14 09:53:59,690 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 09:53:59,690 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 09:53:59,690 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.31 seconds 2025-02-14 09:53:59,690 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:53:59,690 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21583.34 MB 2025-02-14 09:53:59,690 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25975.76 MB 2025-02-14 09:53:59,690 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4392.42 MB 2025-02-14 09:53:59,690 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61190.70 MB 2025-02-14 09:53:59,690 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28447.87 MB 2025-02-14 09:53:59,690 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32742.83 MB 2025-02-14 09:53:59,690 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25975.76 MB 2025-02-14 09:53:59,954 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 09:53:59,954 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 09:53:59,954 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 09:53:59,954 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:53:59,954 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25975.76 MB 2025-02-14 09:53:59,954 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25381.36 MB 2025-02-14 09:53:59,954 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -594.40 MB 2025-02-14 09:53:59,954 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28447.87 MB 2025-02-14 09:53:59,954 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28447.87 MB 2025-02-14 09:53:59,954 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:53:59,954 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27079.54 MB 2025-02-14 09:53:59,972 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8152, cut from 8154 2025-02-14 09:53:59,972 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 09:53:59,978 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 09:53:59,978 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 09:53:59,979 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:53:59,979 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:53:59,979 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25381.36 MB 2025-02-14 09:53:59,979 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33810.48 MB 2025-02-14 09:53:59,979 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8429.12 MB 2025-02-14 09:53:59,979 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28447.87 MB 2025-02-14 09:53:59,979 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38923.14 MB 2025-02-14 09:53:59,979 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10475.27 MB 2025-02-14 09:53:59,979 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33810.48 MB 2025-02-14 09:54:00,129 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7944] 2025-02-14 09:54:00,131 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:54:00,131 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 09:54:00,132 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:54:00,132 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 09:54:00,137 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 09:54:00,138 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:54:00,138 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 09:54:00,138 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 09:54:17,043 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:54:17,043 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 09:54:17,048 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 09:54:17,051 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:54:17,051 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 3124, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 09:54:17,052 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:54:17,052 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 3124, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 09:55:05,189 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 09:55:05,189 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 09:55:05,189 - resource_logging.py:150 - __exit__ - DEBUG - Time: 48.12 seconds 2025-02-14 09:55:05,189 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:55:05,189 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42801.91 MB 2025-02-14 09:55:05,189 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 53858.09 MB 2025-02-14 09:55:05,189 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11056.19 MB 2025-02-14 09:55:05,189 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 69075.99 MB 2025-02-14 09:55:05,189 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 59523.47 MB 2025-02-14 09:55:05,189 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9552.53 MB 2025-02-14 09:55:05,189 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 64913.75 MB 2025-02-14 09:55:05,440 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 09:55:05,441 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 09:55:05,441 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.25 seconds 2025-02-14 09:55:05,441 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:55:05,441 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 53858.09 MB 2025-02-14 09:55:05,441 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40084.01 MB 2025-02-14 09:55:05,441 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -13774.09 MB 2025-02-14 09:55:05,441 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59523.47 MB 2025-02-14 09:55:05,441 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 83074.48 MB 2025-02-14 09:55:05,441 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 23551.02 MB 2025-02-14 09:55:05,441 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 86048.12 MB 2025-02-14 09:55:07,384 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 09:55:07,384 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 09:55:07,384 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-14 09:55:07,384 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:55:07,384 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40084.01 MB 2025-02-14 09:55:07,384 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40614.85 MB 2025-02-14 09:55:07,384 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 09:55:07,384 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 83074.48 MB 2025-02-14 09:55:07,384 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45063.60 MB 2025-02-14 09:55:07,384 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -38010.88 MB 2025-02-14 09:55:07,384 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44594.18 MB 2025-02-14 09:55:07,398 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 09:55:07,398 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 09:55:07,398 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:55:07,398 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:55:07,398 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40614.85 MB 2025-02-14 09:55:07,398 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42504.07 MB 2025-02-14 09:55:07,398 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.22 MB 2025-02-14 09:55:07,398 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45063.60 MB 2025-02-14 09:55:07,398 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46951.04 MB 2025-02-14 09:55:07,398 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 09:55:07,398 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43921.50 MB 2025-02-14 09:55:07,608 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 09:55:07,608 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 09:55:07,608 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 09:55:07,608 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:55:07,608 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42504.07 MB 2025-02-14 09:55:07,608 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44745.93 MB 2025-02-14 09:55:07,608 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 09:55:07,608 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46951.04 MB 2025-02-14 09:55:07,608 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53085.21 MB 2025-02-14 09:55:07,608 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 09:55:07,608 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50291.11 MB 2025-02-14 09:55:07,609 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 09:55:07,609 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 09:55:07,609 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 09:55:07,609 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:55:07,609 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40614.85 MB 2025-02-14 09:55:07,609 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44745.93 MB 2025-02-14 09:55:07,609 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.08 MB 2025-02-14 09:55:07,609 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45063.60 MB 2025-02-14 09:55:07,609 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53085.21 MB 2025-02-14 09:55:07,609 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-14 09:55:07,609 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50291.11 MB 2025-02-14 09:55:07,780 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 09:55:07,780 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 09:55:07,780 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 09:55:07,780 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:55:07,780 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46280.37 MB 2025-02-14 09:55:07,780 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47047.37 MB 2025-02-14 09:55:07,780 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 09:55:07,780 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53085.21 MB 2025-02-14 09:55:07,780 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53498.35 MB 2025-02-14 09:55:07,780 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 09:55:07,780 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47755.16 MB 2025-02-14 09:55:07,799 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 09:55:07,799 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 09:55:07,799 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:55:07,799 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:55:07,799 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47460.26 MB 2025-02-14 09:55:07,799 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47688.94 MB 2025-02-14 09:55:07,799 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.68 MB 2025-02-14 09:55:07,799 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53498.35 MB 2025-02-14 09:55:07,799 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53498.35 MB 2025-02-14 09:55:07,799 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:55:07,799 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47932.95 MB 2025-02-14 09:55:07,800 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 09:55:07,800 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 09:55:07,800 - resource_logging.py:150 - __exit__ - DEBUG - Time: 50.75 seconds 2025-02-14 09:55:07,800 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:55:07,800 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31917.38 MB 2025-02-14 09:55:07,800 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47889.84 MB 2025-02-14 09:55:07,800 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 15972.46 MB 2025-02-14 09:55:07,800 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58189.68 MB 2025-02-14 09:55:07,800 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53498.35 MB 2025-02-14 09:55:07,800 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4691.33 MB 2025-02-14 09:55:07,800 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47932.95 MB 2025-02-14 09:55:08,069 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 09:55:08,069 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 09:55:08,069 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 09:55:08,069 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:55:08,069 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47889.84 MB 2025-02-14 09:55:08,069 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36920.12 MB 2025-02-14 09:55:08,069 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10969.72 MB 2025-02-14 09:55:08,069 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53498.35 MB 2025-02-14 09:55:08,069 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53498.35 MB 2025-02-14 09:55:08,069 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:55:08,069 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50399.36 MB 2025-02-14 09:55:08,087 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8155, cut from 8157 2025-02-14 09:55:08,088 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 09:55:08,093 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 09:55:08,093 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 09:55:08,094 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:55:08,094 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:55:08,094 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36920.12 MB 2025-02-14 09:55:08,094 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45351.59 MB 2025-02-14 09:55:08,094 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8431.46 MB 2025-02-14 09:55:08,094 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53498.35 MB 2025-02-14 09:55:08,094 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57690.55 MB 2025-02-14 09:55:08,094 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4192.21 MB 2025-02-14 09:55:08,094 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45351.59 MB 2025-02-14 09:55:08,256 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7947] 2025-02-14 09:55:08,258 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:55:08,258 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 09:55:08,259 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:55:08,259 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 09:55:08,263 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 09:55:08,264 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:55:08,264 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 09:55:08,264 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 09:55:31,170 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:55:31,170 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 09:55:31,175 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 09:55:31,178 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:55:31,179 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 355, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 09:55:31,179 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:55:31,179 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 355, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 09:55:36,695 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 09:55:36,695 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 09:55:36,695 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.51 seconds 2025-02-14 09:55:36,695 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:55:36,695 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23506.55 MB 2025-02-14 09:55:36,695 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24762.87 MB 2025-02-14 09:55:36,695 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1256.33 MB 2025-02-14 09:55:36,695 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 66074.97 MB 2025-02-14 09:55:36,695 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28349.30 MB 2025-02-14 09:55:36,695 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -37725.67 MB 2025-02-14 09:55:36,695 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33658.20 MB 2025-02-14 09:55:36,719 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 09:55:36,719 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 09:55:36,719 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:55:36,719 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:55:36,719 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24762.87 MB 2025-02-14 09:55:36,719 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25365.13 MB 2025-02-14 09:55:36,719 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 602.25 MB 2025-02-14 09:55:36,719 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28349.30 MB 2025-02-14 09:55:36,719 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33327.94 MB 2025-02-14 09:55:36,719 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4978.64 MB 2025-02-14 09:55:36,719 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29747.10 MB 2025-02-14 09:55:38,404 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 09:55:38,404 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 09:55:38,405 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.68 seconds 2025-02-14 09:55:38,405 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:55:38,405 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25365.13 MB 2025-02-14 09:55:38,405 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25834.92 MB 2025-02-14 09:55:38,405 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 469.79 MB 2025-02-14 09:55:38,405 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33327.94 MB 2025-02-14 09:55:38,405 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30838.62 MB 2025-02-14 09:55:38,405 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2489.32 MB 2025-02-14 09:55:38,405 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29790.37 MB 2025-02-14 09:55:38,417 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 09:55:38,417 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 09:55:38,417 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:55:38,417 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:55:38,417 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25834.92 MB 2025-02-14 09:55:38,417 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27506.88 MB 2025-02-14 09:55:38,417 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1671.95 MB 2025-02-14 09:55:38,417 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30838.62 MB 2025-02-14 09:55:38,417 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32512.15 MB 2025-02-14 09:55:38,417 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1673.53 MB 2025-02-14 09:55:38,417 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28761.30 MB 2025-02-14 09:55:38,602 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 09:55:38,602 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 09:55:38,602 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-14 09:55:38,602 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:55:38,602 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27506.88 MB 2025-02-14 09:55:38,602 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29490.93 MB 2025-02-14 09:55:38,603 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1984.05 MB 2025-02-14 09:55:38,603 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32512.15 MB 2025-02-14 09:55:38,603 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37532.73 MB 2025-02-14 09:55:38,603 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5020.58 MB 2025-02-14 09:55:38,603 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34397.61 MB 2025-02-14 09:55:38,603 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 09:55:38,603 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 09:55:38,603 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 09:55:38,603 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:55:38,603 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25834.92 MB 2025-02-14 09:55:38,603 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29490.93 MB 2025-02-14 09:55:38,603 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3656.01 MB 2025-02-14 09:55:38,603 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30838.62 MB 2025-02-14 09:55:38,603 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37532.73 MB 2025-02-14 09:55:38,603 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6694.11 MB 2025-02-14 09:55:38,603 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34397.61 MB 2025-02-14 09:55:38,770 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 09:55:38,770 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 09:55:38,770 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 09:55:38,770 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:55:38,770 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30848.11 MB 2025-02-14 09:55:38,770 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31526.91 MB 2025-02-14 09:55:38,770 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 678.80 MB 2025-02-14 09:55:38,770 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37532.73 MB 2025-02-14 09:55:38,770 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37895.54 MB 2025-02-14 09:55:38,770 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 362.81 MB 2025-02-14 09:55:38,770 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32153.30 MB 2025-02-14 09:55:38,787 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 09:55:38,787 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 09:55:38,787 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:55:38,787 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:55:38,787 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31892.32 MB 2025-02-14 09:55:38,787 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32098.57 MB 2025-02-14 09:55:38,787 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.25 MB 2025-02-14 09:55:38,787 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37895.54 MB 2025-02-14 09:55:38,787 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37899.73 MB 2025-02-14 09:55:38,787 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-14 09:55:38,787 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32275.39 MB 2025-02-14 09:55:38,788 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 09:55:38,788 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 09:55:38,788 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.61 seconds 2025-02-14 09:55:38,788 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:55:38,788 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22269.70 MB 2025-02-14 09:55:38,788 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32299.64 MB 2025-02-14 09:55:38,788 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 10029.94 MB 2025-02-14 09:55:38,788 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 66074.97 MB 2025-02-14 09:55:38,788 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37899.73 MB 2025-02-14 09:55:38,788 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28175.24 MB 2025-02-14 09:55:38,788 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32299.64 MB 2025-02-14 09:55:39,057 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 09:55:39,057 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 09:55:39,057 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 09:55:39,057 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:55:39,057 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32299.64 MB 2025-02-14 09:55:39,057 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27056.48 MB 2025-02-14 09:55:39,057 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5243.16 MB 2025-02-14 09:55:39,057 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37899.73 MB 2025-02-14 09:55:39,057 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37899.73 MB 2025-02-14 09:55:39,057 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:55:39,057 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35514.58 MB 2025-02-14 09:55:39,075 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 09:55:39,076 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 09:55:39,082 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 09:55:39,082 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 09:55:39,082 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:55:39,082 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:55:39,082 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27056.48 MB 2025-02-14 09:55:39,082 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35495.50 MB 2025-02-14 09:55:39,082 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 09:55:39,082 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37899.73 MB 2025-02-14 09:55:39,082 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48389.69 MB 2025-02-14 09:55:39,082 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 09:55:39,082 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35495.50 MB 2025-02-14 09:55:39,245 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 09:55:39,247 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:55:39,247 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 09:55:39,248 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:55:39,248 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 09:55:39,252 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 09:55:39,253 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:55:39,253 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 09:55:39,254 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 09:56:26,400 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:56:26,400 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 09:56:26,405 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 09:56:26,408 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:56:26,408 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 379, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 09:56:26,409 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:56:26,409 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 379, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 09:56:32,253 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 09:56:32,253 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 09:56:32,253 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.84 seconds 2025-02-14 09:56:32,253 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:56:32,253 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23673.78 MB 2025-02-14 09:56:32,253 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25015.04 MB 2025-02-14 09:56:32,254 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1341.26 MB 2025-02-14 09:56:32,254 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60974.69 MB 2025-02-14 09:56:32,254 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29186.06 MB 2025-02-14 09:56:32,254 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31788.63 MB 2025-02-14 09:56:32,254 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33825.44 MB 2025-02-14 09:56:32,289 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 09:56:32,289 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 09:56:32,289 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 09:56:32,290 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:56:32,290 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25015.04 MB 2025-02-14 09:56:32,290 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25454.26 MB 2025-02-14 09:56:32,290 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 439.21 MB 2025-02-14 09:56:32,290 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29186.06 MB 2025-02-14 09:56:32,290 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32874.95 MB 2025-02-14 09:56:32,290 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3688.89 MB 2025-02-14 09:56:32,290 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29956.30 MB 2025-02-14 09:56:33,976 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 09:56:33,976 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 09:56:33,976 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.68 seconds 2025-02-14 09:56:33,976 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:56:33,976 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25454.26 MB 2025-02-14 09:56:33,976 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25917.41 MB 2025-02-14 09:56:33,976 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 463.16 MB 2025-02-14 09:56:33,976 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32874.95 MB 2025-02-14 09:56:33,976 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30930.89 MB 2025-02-14 09:56:33,976 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1944.06 MB 2025-02-14 09:56:33,976 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29880.53 MB 2025-02-14 09:56:33,989 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 09:56:33,989 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 09:56:33,989 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:56:33,989 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:56:33,989 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25917.41 MB 2025-02-14 09:56:33,989 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27567.69 MB 2025-02-14 09:56:33,989 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1650.28 MB 2025-02-14 09:56:33,989 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30930.89 MB 2025-02-14 09:56:33,989 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32579.26 MB 2025-02-14 09:56:33,989 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1648.36 MB 2025-02-14 09:56:33,989 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28804.40 MB 2025-02-14 09:56:34,177 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 09:56:34,178 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 09:56:34,178 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.19 seconds 2025-02-14 09:56:34,178 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:56:34,178 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27567.69 MB 2025-02-14 09:56:34,178 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29524.51 MB 2025-02-14 09:56:34,178 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1956.81 MB 2025-02-14 09:56:34,178 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32579.26 MB 2025-02-14 09:56:34,178 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37524.34 MB 2025-02-14 09:56:34,178 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4945.08 MB 2025-02-14 09:56:34,178 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34365.03 MB 2025-02-14 09:56:34,178 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 09:56:34,178 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 09:56:34,178 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 09:56:34,178 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:56:34,178 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25917.41 MB 2025-02-14 09:56:34,178 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29524.51 MB 2025-02-14 09:56:34,178 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3607.09 MB 2025-02-14 09:56:34,178 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30930.89 MB 2025-02-14 09:56:34,178 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37524.34 MB 2025-02-14 09:56:34,178 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6593.45 MB 2025-02-14 09:56:34,178 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34365.03 MB 2025-02-14 09:56:34,330 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 09:56:34,330 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 09:56:34,330 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 09:56:34,330 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:56:34,330 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30862.52 MB 2025-02-14 09:56:34,330 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31531.73 MB 2025-02-14 09:56:34,330 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 669.21 MB 2025-02-14 09:56:34,330 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37524.34 MB 2025-02-14 09:56:34,330 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37882.95 MB 2025-02-14 09:56:34,330 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 358.61 MB 2025-02-14 09:56:34,330 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32149.28 MB 2025-02-14 09:56:34,347 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 09:56:34,347 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 09:56:34,347 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:56:34,347 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:56:34,347 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31891.98 MB 2025-02-14 09:56:34,347 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32111.22 MB 2025-02-14 09:56:34,347 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 219.24 MB 2025-02-14 09:56:34,347 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37882.95 MB 2025-02-14 09:56:34,347 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37882.95 MB 2025-02-14 09:56:34,347 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:56:34,347 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32283.48 MB 2025-02-14 09:56:34,348 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 09:56:34,349 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 09:56:34,349 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.94 seconds 2025-02-14 09:56:34,349 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:56:34,349 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22353.32 MB 2025-02-14 09:56:34,349 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32312.29 MB 2025-02-14 09:56:34,349 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 9958.98 MB 2025-02-14 09:56:34,349 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60974.69 MB 2025-02-14 09:56:34,349 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37882.95 MB 2025-02-14 09:56:34,349 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23091.74 MB 2025-02-14 09:56:34,349 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32312.29 MB 2025-02-14 09:56:34,616 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 09:56:34,616 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 09:56:34,616 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 09:56:34,616 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:56:34,616 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32312.29 MB 2025-02-14 09:56:34,617 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35326.33 MB 2025-02-14 09:56:34,617 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-14 09:56:34,617 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37882.95 MB 2025-02-14 09:56:34,617 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37882.95 MB 2025-02-14 09:56:34,617 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:56:34,617 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35627.70 MB 2025-02-14 09:56:34,635 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 09:56:34,635 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-14 09:56:34,641 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 09:56:34,641 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 09:56:34,641 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:56:34,641 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:56:34,641 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27117.02 MB 2025-02-14 09:56:34,641 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35556.05 MB 2025-02-14 09:56:34,641 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 09:56:34,641 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37882.95 MB 2025-02-14 09:56:34,641 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48372.91 MB 2025-02-14 09:56:34,641 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 09:56:34,641 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35556.05 MB 2025-02-14 09:56:34,813 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 09:56:34,814 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:56:34,814 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 09:56:34,815 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:56:34,815 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 09:56:34,820 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 09:56:34,821 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:56:34,821 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 09:56:34,821 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-14 09:56:39,061 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:56:39,061 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 09:56:39,068 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 09:56:39,074 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:56:39,074 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1217, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 09:56:39,076 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:56:39,076 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1217, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 09:56:57,997 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 09:56:57,997 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 09:56:57,997 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.91 seconds 2025-02-14 09:56:57,997 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:56:57,997 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29513.10 MB 2025-02-14 09:56:57,997 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33820.65 MB 2025-02-14 09:56:57,997 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4307.55 MB 2025-02-14 09:56:57,997 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60957.92 MB 2025-02-14 09:56:57,997 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45684.36 MB 2025-02-14 09:56:57,997 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15273.56 MB 2025-02-14 09:56:57,997 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42834.84 MB 2025-02-14 09:56:58,067 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 09:56:58,067 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 09:56:58,067 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 09:56:58,067 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:56:58,067 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33820.65 MB 2025-02-14 09:56:58,067 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30168.80 MB 2025-02-14 09:56:58,067 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3651.85 MB 2025-02-14 09:56:58,067 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45684.36 MB 2025-02-14 09:56:58,067 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52145.68 MB 2025-02-14 09:56:58,067 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6461.33 MB 2025-02-14 09:56:58,067 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46679.13 MB 2025-02-14 09:57:00,002 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 09:57:00,002 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 09:57:00,002 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 09:57:00,002 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:57:00,002 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30168.80 MB 2025-02-14 09:57:00,002 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30699.64 MB 2025-02-14 09:57:00,002 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 09:57:00,002 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52145.68 MB 2025-02-14 09:57:00,002 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41376.81 MB 2025-02-14 09:57:00,002 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10768.88 MB 2025-02-14 09:57:00,002 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34678.97 MB 2025-02-14 09:57:00,015 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 09:57:00,015 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 09:57:00,015 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:57:00,015 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:57:00,015 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30699.64 MB 2025-02-14 09:57:00,015 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32588.99 MB 2025-02-14 09:57:00,015 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.35 MB 2025-02-14 09:57:00,015 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41376.81 MB 2025-02-14 09:57:00,015 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41376.81 MB 2025-02-14 09:57:00,015 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:57:00,015 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34006.42 MB 2025-02-14 09:57:00,230 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 09:57:00,230 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 09:57:00,230 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 09:57:00,230 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:57:00,230 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32588.99 MB 2025-02-14 09:57:00,230 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34830.85 MB 2025-02-14 09:57:00,230 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 09:57:00,230 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41376.81 MB 2025-02-14 09:57:00,230 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44207.96 MB 2025-02-14 09:57:00,230 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-14 09:57:00,230 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40376.03 MB 2025-02-14 09:57:00,231 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 09:57:00,231 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 09:57:00,231 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 09:57:00,231 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:57:00,231 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30699.64 MB 2025-02-14 09:57:00,231 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34830.85 MB 2025-02-14 09:57:00,231 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.21 MB 2025-02-14 09:57:00,231 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41376.81 MB 2025-02-14 09:57:00,231 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44207.96 MB 2025-02-14 09:57:00,231 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-14 09:57:00,231 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40376.03 MB 2025-02-14 09:57:00,400 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 09:57:00,400 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 09:57:00,400 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 09:57:00,400 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:57:00,400 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36365.29 MB 2025-02-14 09:57:00,400 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37132.29 MB 2025-02-14 09:57:00,400 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 09:57:00,400 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44207.96 MB 2025-02-14 09:57:00,400 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44621.10 MB 2025-02-14 09:57:00,400 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 09:57:00,400 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37840.08 MB 2025-02-14 09:57:00,419 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 09:57:00,419 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 09:57:00,419 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:57:00,419 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:57:00,419 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37545.18 MB 2025-02-14 09:57:00,419 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37772.50 MB 2025-02-14 09:57:00,419 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.31 MB 2025-02-14 09:57:00,419 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44621.10 MB 2025-02-14 09:57:00,419 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44621.10 MB 2025-02-14 09:57:00,419 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:57:00,419 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38002.49 MB 2025-02-14 09:57:00,420 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 09:57:00,420 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 09:57:00,420 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.34 seconds 2025-02-14 09:57:00,420 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:57:00,420 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25272.98 MB 2025-02-14 09:57:00,420 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37972.98 MB 2025-02-14 09:57:00,420 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12700.00 MB 2025-02-14 09:57:00,420 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60957.92 MB 2025-02-14 09:57:00,420 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44621.10 MB 2025-02-14 09:57:00,420 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16336.81 MB 2025-02-14 09:57:00,420 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38002.49 MB 2025-02-14 09:57:00,689 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 09:57:00,689 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 09:57:00,689 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 09:57:00,689 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:57:00,689 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37972.98 MB 2025-02-14 09:57:00,689 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30269.42 MB 2025-02-14 09:57:00,689 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7703.56 MB 2025-02-14 09:57:00,689 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44621.10 MB 2025-02-14 09:57:00,689 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44621.10 MB 2025-02-14 09:57:00,689 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:57:00,689 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40477.28 MB 2025-02-14 09:57:00,707 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8138, cut from 8140 2025-02-14 09:57:00,707 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 09:57:00,713 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 09:57:00,713 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 09:57:00,713 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:57:00,713 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:57:00,713 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30269.42 MB 2025-02-14 09:57:00,713 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38683.40 MB 2025-02-14 09:57:00,713 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8413.98 MB 2025-02-14 09:57:00,713 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44621.10 MB 2025-02-14 09:57:00,713 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48802.82 MB 2025-02-14 09:57:00,713 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4181.72 MB 2025-02-14 09:57:00,714 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38683.40 MB 2025-02-14 09:57:00,878 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7930] 2025-02-14 09:57:00,879 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:57:00,879 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 09:57:00,880 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:57:00,880 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 09:57:00,885 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 09:57:00,886 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:57:00,886 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 09:57:00,886 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 09:58:04,255 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:58:04,255 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 09:58:04,260 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 09:58:04,265 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:58:04,265 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 72, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 09:58:04,266 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:58:04,266 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 72, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 09:58:05,422 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 09:58:05,422 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 09:58:05,422 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.15 seconds 2025-02-14 09:58:05,422 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:58:05,422 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21534.56 MB 2025-02-14 09:58:05,422 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21789.36 MB 2025-02-14 09:58:05,422 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 254.80 MB 2025-02-14 09:58:05,422 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61350.08 MB 2025-02-14 09:58:05,422 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26694.65 MB 2025-02-14 09:58:05,422 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34655.44 MB 2025-02-14 09:58:05,422 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30779.44 MB 2025-02-14 09:58:05,425 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 09:58:05,425 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 09:58:05,425 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 09:58:05,425 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:58:05,425 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21789.36 MB 2025-02-14 09:58:05,425 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21912.82 MB 2025-02-14 09:58:05,425 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 123.45 MB 2025-02-14 09:58:05,425 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26694.65 MB 2025-02-14 09:58:05,425 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26694.65 MB 2025-02-14 09:58:05,425 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:58:05,425 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22295.09 MB 2025-02-14 09:58:05,786 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 09:58:05,786 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 09:58:05,786 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.36 seconds 2025-02-14 09:58:05,786 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:58:05,786 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21912.82 MB 2025-02-14 09:58:05,786 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22008.37 MB 2025-02-14 09:58:05,786 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 95.55 MB 2025-02-14 09:58:05,786 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26694.65 MB 2025-02-14 09:58:05,786 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26694.65 MB 2025-02-14 09:58:05,786 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:58:05,786 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25999.36 MB 2025-02-14 09:58:05,791 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 09:58:05,791 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 09:58:05,791 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 09:58:05,791 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:58:05,791 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22008.30 MB 2025-02-14 09:58:05,791 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22348.33 MB 2025-02-14 09:58:05,791 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 340.03 MB 2025-02-14 09:58:05,791 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26694.65 MB 2025-02-14 09:58:05,791 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26694.65 MB 2025-02-14 09:58:05,791 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:58:05,791 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22603.48 MB 2025-02-14 09:58:05,864 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 09:58:05,864 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 09:58:05,864 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 09:58:05,864 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:58:05,864 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22348.33 MB 2025-02-14 09:58:05,864 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22761.36 MB 2025-02-14 09:58:05,864 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 413.02 MB 2025-02-14 09:58:05,864 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26694.65 MB 2025-02-14 09:58:05,864 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26694.65 MB 2025-02-14 09:58:05,864 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:58:05,864 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23749.84 MB 2025-02-14 09:58:05,864 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 09:58:05,864 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 09:58:05,864 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 09:58:05,864 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:58:05,864 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22008.30 MB 2025-02-14 09:58:05,864 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22761.36 MB 2025-02-14 09:58:05,864 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 753.06 MB 2025-02-14 09:58:05,864 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26694.65 MB 2025-02-14 09:58:05,864 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26694.65 MB 2025-02-14 09:58:05,865 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:58:05,865 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23749.84 MB 2025-02-14 09:58:05,902 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 09:58:05,902 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 09:58:05,902 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 09:58:05,902 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:58:05,902 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23160.08 MB 2025-02-14 09:58:05,902 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23333.53 MB 2025-02-14 09:58:05,902 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 173.45 MB 2025-02-14 09:58:05,902 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26694.65 MB 2025-02-14 09:58:05,902 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26801.60 MB 2025-02-14 09:58:05,902 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 106.95 MB 2025-02-14 09:58:05,902 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23460.93 MB 2025-02-14 09:58:05,909 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 09:58:05,909 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 09:58:05,909 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:58:05,909 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:58:05,909 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23443.25 MB 2025-02-14 09:58:05,909 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23616.56 MB 2025-02-14 09:58:05,909 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 173.31 MB 2025-02-14 09:58:05,909 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26801.60 MB 2025-02-14 09:58:05,909 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26801.60 MB 2025-02-14 09:58:05,909 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:58:05,909 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23616.56 MB 2025-02-14 09:58:05,910 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 09:58:05,910 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 09:58:05,910 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.64 seconds 2025-02-14 09:58:05,910 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:58:05,910 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21283.71 MB 2025-02-14 09:58:05,910 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23771.13 MB 2025-02-14 09:58:05,910 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2487.42 MB 2025-02-14 09:58:05,910 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61350.08 MB 2025-02-14 09:58:05,910 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26801.60 MB 2025-02-14 09:58:05,910 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34548.48 MB 2025-02-14 09:58:05,910 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23771.13 MB 2025-02-14 09:58:06,109 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 09:58:06,109 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 09:58:06,109 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 09:58:06,109 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:58:06,110 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23771.13 MB 2025-02-14 09:58:06,110 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24019.60 MB 2025-02-14 09:58:06,110 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 248.48 MB 2025-02-14 09:58:06,110 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26801.60 MB 2025-02-14 09:58:06,110 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26801.60 MB 2025-02-14 09:58:06,110 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:58:06,110 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24251.27 MB 2025-02-14 09:58:06,124 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 6271, cut from 6273 2025-02-14 09:58:06,124 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 09:58:06,129 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 09:58:06,129 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 09:58:06,129 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:58:06,129 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:58:06,129 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24019.60 MB 2025-02-14 09:58:06,129 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30506.42 MB 2025-02-14 09:58:06,129 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6486.82 MB 2025-02-14 09:58:06,129 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26801.60 MB 2025-02-14 09:58:06,129 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34865.15 MB 2025-02-14 09:58:06,129 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8063.55 MB 2025-02-14 09:58:06,129 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30506.42 MB 2025-02-14 09:58:06,254 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 6063] 2025-02-14 09:58:06,255 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:58:06,255 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 09:58:06,256 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:58:06,256 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 09:58:06,261 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 09:58:06,262 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:58:06,262 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 09:58:06,262 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 09:58:20,938 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:58:20,938 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 09:58:20,943 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 09:58:20,946 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:58:20,946 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1361, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 09:58:20,947 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:58:20,947 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1361, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 09:58:41,954 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 09:58:41,954 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 09:58:41,954 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.00 seconds 2025-02-14 09:58:41,954 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:58:41,954 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30516.52 MB 2025-02-14 09:58:41,954 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35333.68 MB 2025-02-14 09:58:41,954 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4817.16 MB 2025-02-14 09:58:41,954 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41315.99 MB 2025-02-14 09:58:41,954 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44413.49 MB 2025-02-14 09:58:41,954 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3097.49 MB 2025-02-14 09:58:41,954 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44292.05 MB 2025-02-14 09:58:42,035 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 09:58:42,035 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 09:58:42,035 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 09:58:42,035 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:58:42,035 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35333.68 MB 2025-02-14 09:58:42,035 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30917.41 MB 2025-02-14 09:58:42,035 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4416.27 MB 2025-02-14 09:58:42,035 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44413.49 MB 2025-02-14 09:58:42,035 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56254.01 MB 2025-02-14 09:58:42,035 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 11840.52 MB 2025-02-14 09:58:42,035 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49713.30 MB 2025-02-14 09:58:43,959 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 09:58:43,959 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 09:58:43,959 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 09:58:43,959 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:58:43,959 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30917.41 MB 2025-02-14 09:58:43,959 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31448.25 MB 2025-02-14 09:58:43,959 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 09:58:43,959 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56254.01 MB 2025-02-14 09:58:43,959 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37786.48 MB 2025-02-14 09:58:43,959 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18467.52 MB 2025-02-14 09:58:43,959 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35427.59 MB 2025-02-14 09:58:43,975 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 09:58:43,975 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 09:58:43,975 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:58:43,975 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:58:43,975 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31448.25 MB 2025-02-14 09:58:43,975 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33337.61 MB 2025-02-14 09:58:43,975 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.35 MB 2025-02-14 09:58:43,975 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37786.48 MB 2025-02-14 09:58:43,975 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39673.92 MB 2025-02-14 09:58:43,975 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 09:58:43,975 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34755.04 MB 2025-02-14 09:58:44,190 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 09:58:44,190 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 09:58:44,190 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 09:58:44,190 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:58:44,190 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33337.61 MB 2025-02-14 09:58:44,190 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35579.46 MB 2025-02-14 09:58:44,190 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 09:58:44,190 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39673.92 MB 2025-02-14 09:58:44,190 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45336.23 MB 2025-02-14 09:58:44,190 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 09:58:44,190 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41124.64 MB 2025-02-14 09:58:44,191 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 09:58:44,191 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 09:58:44,191 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 09:58:44,191 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:58:44,191 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31448.25 MB 2025-02-14 09:58:44,191 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35579.46 MB 2025-02-14 09:58:44,191 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.21 MB 2025-02-14 09:58:44,191 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37786.48 MB 2025-02-14 09:58:44,191 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45336.23 MB 2025-02-14 09:58:44,191 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-14 09:58:44,191 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41124.64 MB 2025-02-14 09:58:44,357 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 09:58:44,358 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 09:58:44,358 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 09:58:44,358 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:58:44,358 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37113.90 MB 2025-02-14 09:58:44,358 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37880.91 MB 2025-02-14 09:58:44,358 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 09:58:44,358 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45336.23 MB 2025-02-14 09:58:44,358 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45749.37 MB 2025-02-14 09:58:44,358 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 09:58:44,358 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38588.70 MB 2025-02-14 09:58:44,376 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 09:58:44,376 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 09:58:44,376 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:58:44,376 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:58:44,376 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38293.80 MB 2025-02-14 09:58:44,376 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38522.73 MB 2025-02-14 09:58:44,376 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.94 MB 2025-02-14 09:58:44,376 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45749.37 MB 2025-02-14 09:58:44,376 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45749.37 MB 2025-02-14 09:58:44,376 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:58:44,376 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38763.71 MB 2025-02-14 09:58:44,377 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 09:58:44,377 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 09:58:44,377 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.43 seconds 2025-02-14 09:58:44,378 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:58:44,378 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25774.68 MB 2025-02-14 09:58:44,378 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38723.58 MB 2025-02-14 09:58:44,378 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12948.90 MB 2025-02-14 09:58:44,378 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41315.99 MB 2025-02-14 09:58:44,378 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45749.37 MB 2025-02-14 09:58:44,378 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4433.38 MB 2025-02-14 09:58:44,378 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38763.71 MB 2025-02-14 09:58:44,646 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 09:58:44,646 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 09:58:44,646 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 09:58:44,646 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:58:44,646 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38723.58 MB 2025-02-14 09:58:44,646 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30777.10 MB 2025-02-14 09:58:44,646 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7946.48 MB 2025-02-14 09:58:44,646 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45749.37 MB 2025-02-14 09:58:44,646 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45749.37 MB 2025-02-14 09:58:44,646 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:58:44,646 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41232.49 MB 2025-02-14 09:58:44,663 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8153, cut from 8155 2025-02-14 09:58:44,664 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 09:58:44,670 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 09:58:44,670 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 09:58:44,670 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:58:44,670 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:58:44,670 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30777.10 MB 2025-02-14 09:58:44,670 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39207.50 MB 2025-02-14 09:58:44,670 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8430.40 MB 2025-02-14 09:58:44,670 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45749.37 MB 2025-02-14 09:58:44,670 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54129.59 MB 2025-02-14 09:58:44,670 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8380.22 MB 2025-02-14 09:58:44,670 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39207.50 MB 2025-02-14 09:58:44,831 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7945] 2025-02-14 09:58:44,832 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:58:44,833 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 09:58:44,833 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:58:44,833 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 09:58:44,838 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 09:58:44,839 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:58:44,839 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 09:58:44,839 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 09:59:19,173 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:59:19,173 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 09:59:19,178 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 09:59:19,182 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:59:19,182 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 257, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 09:59:19,183 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:59:19,183 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 257, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 09:59:23,194 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 09:59:23,194 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 09:59:23,194 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.01 seconds 2025-02-14 09:59:23,194 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:59:23,194 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22823.67 MB 2025-02-14 09:59:23,194 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23733.18 MB 2025-02-14 09:59:23,194 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 909.51 MB 2025-02-14 09:59:23,194 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62509.81 MB 2025-02-14 09:59:23,194 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26694.65 MB 2025-02-14 09:59:23,194 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35815.16 MB 2025-02-14 09:59:23,194 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32748.83 MB 2025-02-14 09:59:23,210 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 09:59:23,211 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 09:59:23,211 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:59:23,211 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:59:23,211 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23733.18 MB 2025-02-14 09:59:23,211 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23759.65 MB 2025-02-14 09:59:23,211 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 26.48 MB 2025-02-14 09:59:23,211 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26694.65 MB 2025-02-14 09:59:23,211 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28439.48 MB 2025-02-14 09:59:23,211 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1744.83 MB 2025-02-14 09:59:23,211 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26518.08 MB 2025-02-14 09:59:24,163 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 09:59:24,163 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 09:59:24,163 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.95 seconds 2025-02-14 09:59:24,163 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:59:24,163 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23759.65 MB 2025-02-14 09:59:24,163 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24022.42 MB 2025-02-14 09:59:24,163 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 262.77 MB 2025-02-14 09:59:24,163 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28439.48 MB 2025-02-14 09:59:24,163 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27736.93 MB 2025-02-14 09:59:24,163 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -702.55 MB 2025-02-14 09:59:24,163 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28016.06 MB 2025-02-14 09:59:24,171 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 09:59:24,171 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 09:59:24,171 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:59:24,171 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:59:24,171 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24022.42 MB 2025-02-14 09:59:24,171 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24957.51 MB 2025-02-14 09:59:24,171 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 935.09 MB 2025-02-14 09:59:24,171 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27736.93 MB 2025-02-14 09:59:24,171 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27736.93 MB 2025-02-14 09:59:24,171 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:59:24,171 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25659.14 MB 2025-02-14 09:59:24,278 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 09:59:24,278 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 09:59:24,278 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 09:59:24,278 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:59:24,278 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24957.51 MB 2025-02-14 09:59:24,278 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26067.26 MB 2025-02-14 09:59:24,278 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1109.75 MB 2025-02-14 09:59:24,278 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27736.93 MB 2025-02-14 09:59:24,278 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30310.14 MB 2025-02-14 09:59:24,278 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2573.21 MB 2025-02-14 09:59:24,278 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28813.75 MB 2025-02-14 09:59:24,279 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 09:59:24,279 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 09:59:24,279 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 09:59:24,279 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:59:24,279 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24022.42 MB 2025-02-14 09:59:24,279 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26067.26 MB 2025-02-14 09:59:24,279 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2044.84 MB 2025-02-14 09:59:24,279 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27736.93 MB 2025-02-14 09:59:24,279 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30310.14 MB 2025-02-14 09:59:24,279 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2573.21 MB 2025-02-14 09:59:24,279 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28813.75 MB 2025-02-14 09:59:24,364 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 09:59:24,364 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 09:59:24,364 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 09:59:24,364 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:59:24,364 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26826.37 MB 2025-02-14 09:59:24,364 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27206.03 MB 2025-02-14 09:59:24,364 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 379.67 MB 2025-02-14 09:59:24,364 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30310.14 MB 2025-02-14 09:59:24,364 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30511.46 MB 2025-02-14 09:59:24,364 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 201.33 MB 2025-02-14 09:59:24,364 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27561.14 MB 2025-02-14 09:59:24,375 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 09:59:24,375 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 09:59:24,375 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:59:24,375 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:59:24,375 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27410.42 MB 2025-02-14 09:59:24,375 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27630.33 MB 2025-02-14 09:59:24,375 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 219.91 MB 2025-02-14 09:59:24,375 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30511.46 MB 2025-02-14 09:59:24,375 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30511.46 MB 2025-02-14 09:59:24,375 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:59:24,375 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27677.38 MB 2025-02-14 09:59:24,376 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 09:59:24,376 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 09:59:24,376 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.19 seconds 2025-02-14 09:59:24,376 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:59:24,376 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21928.26 MB 2025-02-14 09:59:24,376 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27831.28 MB 2025-02-14 09:59:24,376 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5903.02 MB 2025-02-14 09:59:24,376 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62509.81 MB 2025-02-14 09:59:24,376 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30511.46 MB 2025-02-14 09:59:24,376 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31998.35 MB 2025-02-14 09:59:24,376 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27831.28 MB 2025-02-14 09:59:24,643 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 09:59:24,643 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 09:59:24,643 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 09:59:24,643 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:59:24,643 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22965.00 MB 2025-02-14 09:59:24,643 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25977.25 MB 2025-02-14 09:59:24,643 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3012.25 MB 2025-02-14 09:59:24,643 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30511.46 MB 2025-02-14 09:59:24,643 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30511.46 MB 2025-02-14 09:59:24,643 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:59:24,643 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26278.43 MB 2025-02-14 09:59:24,661 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8157, cut from 8159 2025-02-14 09:59:24,662 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 09:59:24,668 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 09:59:24,668 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 09:59:24,668 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:59:24,668 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:59:24,668 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25977.25 MB 2025-02-14 09:59:24,668 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34411.86 MB 2025-02-14 09:59:24,668 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8434.62 MB 2025-02-14 09:59:24,668 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30511.46 MB 2025-02-14 09:59:24,668 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40993.03 MB 2025-02-14 09:59:24,668 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10481.57 MB 2025-02-14 09:59:24,668 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34411.86 MB 2025-02-14 09:59:24,831 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7949] 2025-02-14 09:59:24,832 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:59:24,832 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 09:59:24,833 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:59:24,833 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 09:59:24,838 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 09:59:24,839 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:59:24,839 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 09:59:24,839 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 09:59:36,437 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:59:36,437 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 09:59:36,445 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 09:59:36,452 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:59:36,452 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 677, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 09:59:36,454 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:59:36,454 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 677, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 09:59:46,970 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 09:59:46,970 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 09:59:46,970 - resource_logging.py:150 - __exit__ - DEBUG - Time: 10.51 seconds 2025-02-14 09:59:46,970 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:59:46,970 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25750.30 MB 2025-02-14 09:59:46,970 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28146.16 MB 2025-02-14 09:59:46,970 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2395.87 MB 2025-02-14 09:59:46,970 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49377.44 MB 2025-02-14 09:59:46,970 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35381.05 MB 2025-02-14 09:59:46,970 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13996.39 MB 2025-02-14 09:59:46,970 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37033.61 MB 2025-02-14 09:59:47,013 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 09:59:47,013 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 09:59:47,013 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-14 09:59:47,013 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:59:47,013 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28146.16 MB 2025-02-14 09:59:47,013 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27361.51 MB 2025-02-14 09:59:47,013 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -784.65 MB 2025-02-14 09:59:47,013 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35381.05 MB 2025-02-14 09:59:47,013 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41435.53 MB 2025-02-14 09:59:47,013 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6054.48 MB 2025-02-14 09:59:47,013 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36693.76 MB 2025-02-14 09:59:48,922 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 09:59:48,922 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 09:59:48,922 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 09:59:48,922 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:59:48,922 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27361.51 MB 2025-02-14 09:59:48,922 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27892.35 MB 2025-02-14 09:59:48,922 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 09:59:48,922 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41435.53 MB 2025-02-14 09:59:48,922 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34399.58 MB 2025-02-14 09:59:48,922 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7035.94 MB 2025-02-14 09:59:48,922 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31871.68 MB 2025-02-14 09:59:48,935 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 09:59:48,935 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 09:59:48,935 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 09:59:48,935 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:59:48,935 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27892.35 MB 2025-02-14 09:59:48,935 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29781.70 MB 2025-02-14 09:59:48,935 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.35 MB 2025-02-14 09:59:48,935 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34399.58 MB 2025-02-14 09:59:48,935 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35343.30 MB 2025-02-14 09:59:48,935 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 09:59:48,935 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31199.13 MB 2025-02-14 09:59:49,145 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 09:59:49,145 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 09:59:49,145 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 09:59:49,145 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:59:49,145 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29781.70 MB 2025-02-14 09:59:49,145 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32023.56 MB 2025-02-14 09:59:49,145 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 09:59:49,146 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35343.30 MB 2025-02-14 09:59:49,146 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41005.61 MB 2025-02-14 09:59:49,146 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 09:59:49,146 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37568.74 MB 2025-02-14 09:59:49,146 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 09:59:49,146 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 09:59:49,146 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 09:59:49,147 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:59:49,147 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27892.35 MB 2025-02-14 09:59:49,147 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32023.56 MB 2025-02-14 09:59:49,147 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.21 MB 2025-02-14 09:59:49,147 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34399.58 MB 2025-02-14 09:59:49,147 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41005.61 MB 2025-02-14 09:59:49,147 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 09:59:49,147 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37568.74 MB 2025-02-14 09:59:49,313 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 09:59:49,313 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 09:59:49,313 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 09:59:49,313 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:59:49,313 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33558.00 MB 2025-02-14 09:59:49,313 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34325.00 MB 2025-02-14 09:59:49,313 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 09:59:49,313 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41005.61 MB 2025-02-14 09:59:49,313 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41418.75 MB 2025-02-14 09:59:49,313 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 09:59:49,313 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35032.79 MB 2025-02-14 09:59:49,332 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 09:59:49,332 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 09:59:49,332 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:59:49,332 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:59:49,332 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34737.89 MB 2025-02-14 09:59:49,332 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34970.65 MB 2025-02-14 09:59:49,332 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 232.76 MB 2025-02-14 09:59:49,332 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41418.75 MB 2025-02-14 09:59:49,332 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41418.75 MB 2025-02-14 09:59:49,332 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:59:49,332 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35149.42 MB 2025-02-14 09:59:49,333 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 09:59:49,333 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 09:59:49,333 - resource_logging.py:150 - __exit__ - DEBUG - Time: 12.88 seconds 2025-02-14 09:59:49,333 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:59:49,333 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23391.57 MB 2025-02-14 09:59:49,333 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35171.72 MB 2025-02-14 09:59:49,333 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11780.15 MB 2025-02-14 09:59:49,333 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49377.44 MB 2025-02-14 09:59:49,333 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41418.75 MB 2025-02-14 09:59:49,333 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7958.69 MB 2025-02-14 09:59:49,333 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35171.72 MB 2025-02-14 09:59:49,600 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 09:59:49,601 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 09:59:49,601 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 09:59:49,601 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:59:49,601 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35171.72 MB 2025-02-14 09:59:49,601 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28396.86 MB 2025-02-14 09:59:49,601 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6774.86 MB 2025-02-14 09:59:49,601 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41418.75 MB 2025-02-14 09:59:49,601 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41418.75 MB 2025-02-14 09:59:49,601 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 09:59:49,601 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37683.39 MB 2025-02-14 09:59:49,619 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 09:59:49,619 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 09:59:49,625 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 09:59:49,625 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 09:59:49,625 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 09:59:49,625 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 09:59:49,625 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28396.86 MB 2025-02-14 09:59:49,625 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36835.89 MB 2025-02-14 09:59:49,625 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 09:59:49,625 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41418.75 MB 2025-02-14 09:59:49,625 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49809.46 MB 2025-02-14 09:59:49,625 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 09:59:49,625 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36835.89 MB 2025-02-14 09:59:49,786 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 09:59:49,788 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:59:49,788 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 09:59:49,789 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:59:49,789 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 09:59:49,793 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 09:59:49,795 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 09:59:49,795 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 09:59:49,795 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 10:00:41,214 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:00:41,214 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 10:00:41,219 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 10:00:41,223 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:00:41,223 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 182, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 10:00:41,224 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:00:41,224 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 182, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 10:00:44,068 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 10:00:44,068 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 10:00:44,068 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.84 seconds 2025-02-14 10:00:44,068 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:00:44,068 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22301.06 MB 2025-02-14 10:00:44,068 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22945.14 MB 2025-02-14 10:00:44,068 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 644.09 MB 2025-02-14 10:00:44,068 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62394.47 MB 2025-02-14 10:00:44,068 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26696.74 MB 2025-02-14 10:00:44,068 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35697.72 MB 2025-02-14 10:00:44,068 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31773.23 MB 2025-02-14 10:00:44,079 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 10:00:44,079 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 10:00:44,079 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:00:44,079 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:00:44,079 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22945.14 MB 2025-02-14 10:00:44,079 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22660.25 MB 2025-02-14 10:00:44,079 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -284.90 MB 2025-02-14 10:00:44,079 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26696.74 MB 2025-02-14 10:00:44,079 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26696.74 MB 2025-02-14 10:00:44,079 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:00:44,079 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24328.91 MB 2025-02-14 10:00:44,546 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 10:00:44,546 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 10:00:44,546 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.47 seconds 2025-02-14 10:00:44,546 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:00:44,546 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22660.25 MB 2025-02-14 10:00:44,546 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22788.98 MB 2025-02-14 10:00:44,546 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 128.73 MB 2025-02-14 10:00:44,546 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26696.74 MB 2025-02-14 10:00:44,546 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26696.74 MB 2025-02-14 10:00:44,546 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:00:44,546 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26746.79 MB 2025-02-14 10:00:44,553 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 10:00:44,553 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 10:00:44,553 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 10:00:44,553 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:00:44,553 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22788.91 MB 2025-02-14 10:00:44,553 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23247.01 MB 2025-02-14 10:00:44,553 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 458.10 MB 2025-02-14 10:00:44,553 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26696.74 MB 2025-02-14 10:00:44,553 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26696.74 MB 2025-02-14 10:00:44,553 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:00:44,553 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23590.74 MB 2025-02-14 10:00:44,651 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 10:00:44,651 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 10:00:44,651 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 10:00:44,651 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:00:44,651 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23247.01 MB 2025-02-14 10:00:44,651 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23803.62 MB 2025-02-14 10:00:44,651 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 556.61 MB 2025-02-14 10:00:44,651 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26696.74 MB 2025-02-14 10:00:44,651 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26696.74 MB 2025-02-14 10:00:44,651 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:00:44,651 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25135.35 MB 2025-02-14 10:00:44,652 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 10:00:44,652 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 10:00:44,652 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 10:00:44,652 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:00:44,652 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22788.91 MB 2025-02-14 10:00:44,652 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23803.62 MB 2025-02-14 10:00:44,652 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1014.71 MB 2025-02-14 10:00:44,652 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26696.74 MB 2025-02-14 10:00:44,652 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26696.74 MB 2025-02-14 10:00:44,652 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:00:44,652 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25135.35 MB 2025-02-14 10:00:44,702 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 10:00:44,702 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 10:00:44,702 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-14 10:00:44,702 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:00:44,702 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24340.79 MB 2025-02-14 10:00:44,702 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24574.46 MB 2025-02-14 10:00:44,702 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 233.68 MB 2025-02-14 10:00:44,702 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26696.74 MB 2025-02-14 10:00:44,702 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26841.45 MB 2025-02-14 10:00:44,702 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 144.70 MB 2025-02-14 10:00:44,702 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24746.10 MB 2025-02-14 10:00:44,708 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 10:00:44,708 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 10:00:44,708 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 10:00:44,708 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:00:44,708 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24722.27 MB 2025-02-14 10:00:44,708 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24947.80 MB 2025-02-14 10:00:44,708 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 225.53 MB 2025-02-14 10:00:44,708 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26841.45 MB 2025-02-14 10:00:44,708 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26841.45 MB 2025-02-14 10:00:44,708 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:00:44,708 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24947.80 MB 2025-02-14 10:00:44,709 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 10:00:44,709 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 10:00:44,709 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.48 seconds 2025-02-14 10:00:44,709 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:00:44,709 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21666.95 MB 2025-02-14 10:00:44,709 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25148.60 MB 2025-02-14 10:00:44,709 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3481.65 MB 2025-02-14 10:00:44,709 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62394.47 MB 2025-02-14 10:00:44,710 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26841.45 MB 2025-02-14 10:00:44,710 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35553.02 MB 2025-02-14 10:00:44,710 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25148.60 MB 2025-02-14 10:00:44,974 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 10:00:44,974 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 10:00:44,974 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 10:00:44,974 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:00:44,974 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22316.03 MB 2025-02-14 10:00:44,974 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25326.82 MB 2025-02-14 10:00:44,974 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3010.79 MB 2025-02-14 10:00:44,974 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26841.45 MB 2025-02-14 10:00:44,974 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26841.45 MB 2025-02-14 10:00:44,974 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:00:44,974 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25627.79 MB 2025-02-14 10:00:44,992 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8151, cut from 8153 2025-02-14 10:00:44,992 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 10:00:44,998 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 10:00:44,998 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 10:00:44,998 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:00:44,998 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:00:44,998 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25326.82 MB 2025-02-14 10:00:44,998 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33754.16 MB 2025-02-14 10:00:44,998 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8427.34 MB 2025-02-14 10:00:44,998 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26841.45 MB 2025-02-14 10:00:44,998 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37316.72 MB 2025-02-14 10:00:44,999 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10475.27 MB 2025-02-14 10:00:44,999 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33754.16 MB 2025-02-14 10:00:45,164 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7943] 2025-02-14 10:00:45,166 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:00:45,166 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 10:00:45,167 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:00:45,167 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 10:00:45,171 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 10:00:45,173 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:00:45,173 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 10:00:45,173 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 10:01:02,539 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:01:02,539 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 10:01:02,544 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 10:01:02,547 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:01:02,548 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1316, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 10:01:02,548 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:01:02,549 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1316, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 10:01:22,754 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 10:01:22,754 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 10:01:22,754 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.20 seconds 2025-02-14 10:01:22,754 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:01:22,754 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30202.95 MB 2025-02-14 10:01:22,754 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34860.73 MB 2025-02-14 10:01:22,754 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4657.77 MB 2025-02-14 10:01:22,754 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45696.94 MB 2025-02-14 10:01:22,754 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48110.76 MB 2025-02-14 10:01:22,754 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2413.82 MB 2025-02-14 10:01:22,754 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43751.18 MB 2025-02-14 10:01:22,816 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 10:01:22,816 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 10:01:22,816 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 10:01:22,816 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:01:22,816 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34860.73 MB 2025-02-14 10:01:22,816 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30683.47 MB 2025-02-14 10:01:22,816 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4177.26 MB 2025-02-14 10:01:22,816 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48110.76 MB 2025-02-14 10:01:22,816 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50748.98 MB 2025-02-14 10:01:22,816 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2638.22 MB 2025-02-14 10:01:22,816 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44331.82 MB 2025-02-14 10:01:24,722 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 10:01:24,722 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 10:01:24,722 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.90 seconds 2025-02-14 10:01:24,722 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:01:24,722 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30683.47 MB 2025-02-14 10:01:24,722 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31214.31 MB 2025-02-14 10:01:24,722 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 10:01:24,722 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50748.98 MB 2025-02-14 10:01:24,722 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39262.88 MB 2025-02-14 10:01:24,722 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11486.10 MB 2025-02-14 10:01:24,722 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35193.65 MB 2025-02-14 10:01:24,736 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 10:01:24,736 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 10:01:24,736 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:01:24,736 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:01:24,736 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31214.31 MB 2025-02-14 10:01:24,736 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33103.67 MB 2025-02-14 10:01:24,736 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.35 MB 2025-02-14 10:01:24,736 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39262.88 MB 2025-02-14 10:01:24,736 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39262.88 MB 2025-02-14 10:01:24,736 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:01:24,736 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34521.09 MB 2025-02-14 10:01:24,947 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 10:01:24,947 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 10:01:24,947 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 10:01:24,947 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:01:24,947 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33103.67 MB 2025-02-14 10:01:24,947 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35345.52 MB 2025-02-14 10:01:24,947 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 10:01:24,947 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39262.88 MB 2025-02-14 10:01:24,947 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44925.19 MB 2025-02-14 10:01:24,947 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 10:01:24,947 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40890.70 MB 2025-02-14 10:01:24,948 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 10:01:24,948 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 10:01:24,948 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 10:01:24,948 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:01:24,948 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31214.31 MB 2025-02-14 10:01:24,948 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35345.52 MB 2025-02-14 10:01:24,948 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.21 MB 2025-02-14 10:01:24,948 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39262.88 MB 2025-02-14 10:01:24,948 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44925.19 MB 2025-02-14 10:01:24,948 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 10:01:24,948 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40890.70 MB 2025-02-14 10:01:25,117 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 10:01:25,117 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 10:01:25,117 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 10:01:25,117 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:01:25,117 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36879.96 MB 2025-02-14 10:01:25,117 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37646.97 MB 2025-02-14 10:01:25,117 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 10:01:25,117 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44925.19 MB 2025-02-14 10:01:25,117 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45340.43 MB 2025-02-14 10:01:25,117 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 10:01:25,117 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38354.75 MB 2025-02-14 10:01:25,136 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 10:01:25,136 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 10:01:25,136 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:01:25,136 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:01:25,136 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38059.85 MB 2025-02-14 10:01:25,136 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38289.69 MB 2025-02-14 10:01:25,136 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.83 MB 2025-02-14 10:01:25,136 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45340.43 MB 2025-02-14 10:01:25,136 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45340.43 MB 2025-02-14 10:01:25,136 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:01:25,136 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38500.84 MB 2025-02-14 10:01:25,137 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 10:01:25,137 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 10:01:25,137 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.59 seconds 2025-02-14 10:01:25,137 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:01:25,137 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25617.90 MB 2025-02-14 10:01:25,137 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38490.76 MB 2025-02-14 10:01:25,137 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12872.86 MB 2025-02-14 10:01:25,137 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45696.94 MB 2025-02-14 10:01:25,137 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45340.43 MB 2025-02-14 10:01:25,137 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -356.52 MB 2025-02-14 10:01:25,137 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38500.84 MB 2025-02-14 10:01:25,407 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 10:01:25,407 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 10:01:25,407 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 10:01:25,407 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:01:25,407 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38490.76 MB 2025-02-14 10:01:25,407 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30623.19 MB 2025-02-14 10:01:25,407 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7867.57 MB 2025-02-14 10:01:25,407 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45340.43 MB 2025-02-14 10:01:25,407 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45340.43 MB 2025-02-14 10:01:25,407 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:01:25,407 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41002.43 MB 2025-02-14 10:01:25,425 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 10:01:25,426 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 10:01:25,432 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 10:01:25,432 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 10:01:25,432 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:01:25,432 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:01:25,432 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30623.19 MB 2025-02-14 10:01:25,432 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39062.21 MB 2025-02-14 10:01:25,432 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 10:01:25,432 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45340.43 MB 2025-02-14 10:01:25,432 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53731.13 MB 2025-02-14 10:01:25,432 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 10:01:25,432 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39062.21 MB 2025-02-14 10:01:25,595 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 10:01:25,597 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:01:25,597 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 10:01:25,598 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:01:25,598 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 10:01:25,603 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 10:01:25,604 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:01:25,604 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 10:01:25,604 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 10:01:34,791 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:01:34,792 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 10:01:34,796 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 10:01:34,800 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:01:34,800 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 319, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 10:01:34,801 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:01:34,801 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 319, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 10:01:39,744 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 10:01:39,744 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 10:01:39,744 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.94 seconds 2025-02-14 10:01:39,744 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:01:39,744 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23255.70 MB 2025-02-14 10:01:39,744 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24384.62 MB 2025-02-14 10:01:39,744 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1128.92 MB 2025-02-14 10:01:39,744 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 66316.14 MB 2025-02-14 10:01:39,744 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27638.37 MB 2025-02-14 10:01:39,744 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -38677.77 MB 2025-02-14 10:01:39,744 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33262.78 MB 2025-02-14 10:01:39,762 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 10:01:39,762 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 10:01:39,762 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:01:39,762 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:01:39,762 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24384.62 MB 2025-02-14 10:01:39,762 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24123.87 MB 2025-02-14 10:01:39,762 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -260.75 MB 2025-02-14 10:01:39,762 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27638.37 MB 2025-02-14 10:01:39,762 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29085.40 MB 2025-02-14 10:01:39,762 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1447.03 MB 2025-02-14 10:01:39,762 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27303.10 MB 2025-02-14 10:01:40,746 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 10:01:40,746 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 10:01:40,746 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.98 seconds 2025-02-14 10:01:40,746 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:01:40,746 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24123.87 MB 2025-02-14 10:01:40,746 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24394.59 MB 2025-02-14 10:01:40,746 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 270.73 MB 2025-02-14 10:01:40,746 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29085.40 MB 2025-02-14 10:01:40,746 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28764.54 MB 2025-02-14 10:01:40,746 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -320.86 MB 2025-02-14 10:01:40,746 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28380.28 MB 2025-02-14 10:01:40,754 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 10:01:40,754 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 10:01:40,754 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:01:40,754 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:01:40,754 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24394.59 MB 2025-02-14 10:01:40,754 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25358.02 MB 2025-02-14 10:01:40,754 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 963.43 MB 2025-02-14 10:01:40,754 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28764.54 MB 2025-02-14 10:01:40,754 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28764.54 MB 2025-02-14 10:01:40,754 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:01:40,754 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26081.31 MB 2025-02-14 10:01:40,863 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 10:01:40,863 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 10:01:40,863 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 10:01:40,863 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:01:40,863 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25358.02 MB 2025-02-14 10:01:40,863 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26501.40 MB 2025-02-14 10:01:40,863 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1143.38 MB 2025-02-14 10:01:40,863 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28764.54 MB 2025-02-14 10:01:40,863 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31176.26 MB 2025-02-14 10:01:40,863 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2411.72 MB 2025-02-14 10:01:40,863 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29333.54 MB 2025-02-14 10:01:40,864 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 10:01:40,864 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 10:01:40,864 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 10:01:40,864 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:01:40,864 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24394.59 MB 2025-02-14 10:01:40,864 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26501.40 MB 2025-02-14 10:01:40,864 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2106.81 MB 2025-02-14 10:01:40,864 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28764.54 MB 2025-02-14 10:01:40,864 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31176.26 MB 2025-02-14 10:01:40,864 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2411.72 MB 2025-02-14 10:01:40,864 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29333.54 MB 2025-02-14 10:01:40,950 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 10:01:40,950 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 10:01:40,950 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 10:01:40,950 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:01:40,950 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27283.51 MB 2025-02-14 10:01:40,950 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27676.12 MB 2025-02-14 10:01:40,950 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 392.62 MB 2025-02-14 10:01:40,950 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31176.26 MB 2025-02-14 10:01:40,950 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31383.88 MB 2025-02-14 10:01:40,950 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 207.62 MB 2025-02-14 10:01:40,950 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28037.65 MB 2025-02-14 10:01:40,961 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 10:01:40,961 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 10:01:40,961 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:01:40,961 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:01:40,961 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27886.70 MB 2025-02-14 10:01:40,961 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28098.58 MB 2025-02-14 10:01:40,961 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 211.87 MB 2025-02-14 10:01:40,961 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31383.88 MB 2025-02-14 10:01:40,961 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31383.88 MB 2025-02-14 10:01:40,961 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:01:40,961 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28141.16 MB 2025-02-14 10:01:40,962 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 10:01:40,962 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 10:01:40,962 - resource_logging.py:150 - __exit__ - DEBUG - Time: 6.16 seconds 2025-02-14 10:01:40,962 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:01:40,962 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22144.27 MB 2025-02-14 10:01:40,962 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28299.65 MB 2025-02-14 10:01:40,962 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6155.38 MB 2025-02-14 10:01:40,962 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 66316.14 MB 2025-02-14 10:01:40,963 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31383.88 MB 2025-02-14 10:01:40,963 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34932.26 MB 2025-02-14 10:01:40,963 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28299.65 MB 2025-02-14 10:01:41,230 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 10:01:41,230 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 10:01:41,230 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 10:01:41,230 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:01:41,230 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23210.57 MB 2025-02-14 10:01:41,230 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26225.39 MB 2025-02-14 10:01:41,230 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.82 MB 2025-02-14 10:01:41,230 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31383.88 MB 2025-02-14 10:01:41,230 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31383.88 MB 2025-02-14 10:01:41,230 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:01:41,230 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26526.76 MB 2025-02-14 10:01:41,248 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 10:01:41,248 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 10:01:41,255 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 10:01:41,255 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 10:01:41,255 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:01:41,255 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:01:41,255 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26225.39 MB 2025-02-14 10:01:41,255 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34664.41 MB 2025-02-14 10:01:41,255 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 10:01:41,255 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31383.88 MB 2025-02-14 10:01:41,255 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41873.83 MB 2025-02-14 10:01:41,255 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 10:01:41,255 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34664.41 MB 2025-02-14 10:01:41,418 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 10:01:41,419 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:01:41,419 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 10:01:41,420 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:01:41,420 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 10:01:41,425 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 10:01:41,426 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:01:41,426 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 10:01:41,426 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 10:02:42,891 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:02:42,891 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 10:02:42,897 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 10:02:42,900 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:02:42,900 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 182, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 10:02:42,901 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:02:42,901 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 182, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 10:02:45,694 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 10:02:45,694 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 10:02:45,694 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.79 seconds 2025-02-14 10:02:45,694 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:02:45,694 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22301.06 MB 2025-02-14 10:02:45,694 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22945.14 MB 2025-02-14 10:02:45,694 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 644.09 MB 2025-02-14 10:02:45,694 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54458.84 MB 2025-02-14 10:02:45,694 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28038.92 MB 2025-02-14 10:02:45,694 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26419.92 MB 2025-02-14 10:02:45,694 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31773.23 MB 2025-02-14 10:02:45,707 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 10:02:45,707 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 10:02:45,707 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:02:45,707 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:02:45,707 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22945.14 MB 2025-02-14 10:02:45,707 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23130.79 MB 2025-02-14 10:02:45,707 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 185.64 MB 2025-02-14 10:02:45,707 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28038.92 MB 2025-02-14 10:02:45,707 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28038.92 MB 2025-02-14 10:02:45,707 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:02:45,707 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25270.00 MB 2025-02-14 10:02:46,490 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 10:02:46,490 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 10:02:46,490 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.78 seconds 2025-02-14 10:02:46,490 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:02:46,490 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23130.79 MB 2025-02-14 10:02:46,490 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23348.43 MB 2025-02-14 10:02:46,490 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 217.65 MB 2025-02-14 10:02:46,490 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28038.92 MB 2025-02-14 10:02:46,490 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27636.27 MB 2025-02-14 10:02:46,490 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -402.65 MB 2025-02-14 10:02:46,490 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27302.26 MB 2025-02-14 10:02:46,498 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 10:02:46,498 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 10:02:46,498 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 10:02:46,498 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:02:46,498 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23348.37 MB 2025-02-14 10:02:46,498 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24122.89 MB 2025-02-14 10:02:46,498 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 774.52 MB 2025-02-14 10:02:46,498 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27636.27 MB 2025-02-14 10:02:46,498 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27636.27 MB 2025-02-14 10:02:46,498 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:02:46,498 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24704.04 MB 2025-02-14 10:02:46,586 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 10:02:46,586 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 10:02:46,586 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 10:02:46,586 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:02:46,586 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24122.89 MB 2025-02-14 10:02:46,586 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25042.09 MB 2025-02-14 10:02:46,586 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 919.20 MB 2025-02-14 10:02:46,586 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27636.27 MB 2025-02-14 10:02:46,586 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28995.22 MB 2025-02-14 10:02:46,586 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1358.95 MB 2025-02-14 10:02:46,586 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27318.35 MB 2025-02-14 10:02:46,586 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 10:02:46,586 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 10:02:46,586 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 10:02:46,587 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:02:46,587 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23348.37 MB 2025-02-14 10:02:46,587 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25042.09 MB 2025-02-14 10:02:46,587 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1693.72 MB 2025-02-14 10:02:46,587 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27636.27 MB 2025-02-14 10:02:46,587 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28995.22 MB 2025-02-14 10:02:46,587 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1358.95 MB 2025-02-14 10:02:46,587 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27318.35 MB 2025-02-14 10:02:46,654 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 10:02:46,654 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 10:02:46,654 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 10:02:46,654 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:02:46,654 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25670.84 MB 2025-02-14 10:02:46,654 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25985.31 MB 2025-02-14 10:02:46,654 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 314.47 MB 2025-02-14 10:02:46,654 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28995.22 MB 2025-02-14 10:02:46,654 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29163.00 MB 2025-02-14 10:02:46,654 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 167.77 MB 2025-02-14 10:02:46,655 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26283.84 MB 2025-02-14 10:02:46,664 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 10:02:46,664 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 10:02:46,664 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:02:46,664 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:02:46,664 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26154.60 MB 2025-02-14 10:02:46,664 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26366.92 MB 2025-02-14 10:02:46,664 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 212.32 MB 2025-02-14 10:02:46,664 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29163.00 MB 2025-02-14 10:02:46,664 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29163.00 MB 2025-02-14 10:02:46,664 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:02:46,664 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26396.88 MB 2025-02-14 10:02:46,665 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 10:02:46,665 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 10:02:46,665 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.76 seconds 2025-02-14 10:02:46,665 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:02:46,665 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21666.95 MB 2025-02-14 10:02:46,665 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26567.63 MB 2025-02-14 10:02:46,665 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4900.67 MB 2025-02-14 10:02:46,665 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54458.84 MB 2025-02-14 10:02:46,665 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29163.00 MB 2025-02-14 10:02:46,665 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25295.85 MB 2025-02-14 10:02:46,665 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26567.63 MB 2025-02-14 10:02:46,931 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 10:02:46,931 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 10:02:46,931 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 10:02:46,931 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:02:46,931 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26567.63 MB 2025-02-14 10:02:46,931 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25551.54 MB 2025-02-14 10:02:46,931 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1016.09 MB 2025-02-14 10:02:46,931 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29163.00 MB 2025-02-14 10:02:46,931 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29163.00 MB 2025-02-14 10:02:46,931 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:02:46,931 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27269.61 MB 2025-02-14 10:02:46,949 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8147, cut from 8149 2025-02-14 10:02:46,950 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 10:02:46,956 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 10:02:46,956 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 10:02:46,956 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:02:46,956 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:02:46,956 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25551.54 MB 2025-02-14 10:02:46,956 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33974.74 MB 2025-02-14 10:02:46,956 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8423.21 MB 2025-02-14 10:02:46,956 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29163.00 MB 2025-02-14 10:02:46,956 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39634.08 MB 2025-02-14 10:02:46,956 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10471.08 MB 2025-02-14 10:02:46,956 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33974.74 MB 2025-02-14 10:02:47,106 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7939] 2025-02-14 10:02:47,108 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:02:47,108 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 10:02:47,109 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:02:47,109 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 10:02:47,113 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 10:02:47,114 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:02:47,114 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 10:02:47,114 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 10:02:57,203 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:02:57,203 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 10:02:57,210 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 10:02:57,215 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:02:57,216 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1470, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 10:02:57,217 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:02:57,218 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1470, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 10:03:19,821 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 10:03:19,822 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 10:03:19,822 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.60 seconds 2025-02-14 10:03:19,822 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:03:19,822 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31276.05 MB 2025-02-14 10:03:19,822 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36479.08 MB 2025-02-14 10:03:19,822 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5203.03 MB 2025-02-14 10:03:19,822 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48010.10 MB 2025-02-14 10:03:19,822 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48647.63 MB 2025-02-14 10:03:19,822 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 637.53 MB 2025-02-14 10:03:19,822 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45277.27 MB 2025-02-14 10:03:19,906 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 10:03:19,906 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 10:03:19,906 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 10:03:19,906 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:03:19,906 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36479.08 MB 2025-02-14 10:03:19,906 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31484.07 MB 2025-02-14 10:03:19,906 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4995.01 MB 2025-02-14 10:03:19,906 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48647.63 MB 2025-02-14 10:03:19,906 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58707.67 MB 2025-02-14 10:03:19,906 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10060.04 MB 2025-02-14 10:03:19,906 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51655.92 MB 2025-02-14 10:03:21,811 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 10:03:21,811 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 10:03:21,811 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.90 seconds 2025-02-14 10:03:21,811 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:03:21,811 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31484.07 MB 2025-02-14 10:03:21,811 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32014.91 MB 2025-02-14 10:03:21,811 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 10:03:21,811 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58707.67 MB 2025-02-14 10:03:21,811 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43444.60 MB 2025-02-14 10:03:21,811 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15263.07 MB 2025-02-14 10:03:21,811 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35994.24 MB 2025-02-14 10:03:21,824 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 10:03:21,824 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 10:03:21,824 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:03:21,824 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:03:21,824 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32014.91 MB 2025-02-14 10:03:21,824 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33904.26 MB 2025-02-14 10:03:21,824 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.35 MB 2025-02-14 10:03:21,824 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43444.60 MB 2025-02-14 10:03:21,824 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43444.60 MB 2025-02-14 10:03:21,824 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:03:21,824 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35321.69 MB 2025-02-14 10:03:22,037 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 10:03:22,037 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 10:03:22,037 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 10:03:22,037 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:03:22,037 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33904.26 MB 2025-02-14 10:03:22,037 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36146.12 MB 2025-02-14 10:03:22,037 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 10:03:22,037 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43444.60 MB 2025-02-14 10:03:22,037 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47219.47 MB 2025-02-14 10:03:22,037 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 10:03:22,037 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41691.30 MB 2025-02-14 10:03:22,037 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 10:03:22,037 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 10:03:22,037 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 10:03:22,037 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:03:22,037 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32014.91 MB 2025-02-14 10:03:22,037 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36146.12 MB 2025-02-14 10:03:22,037 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.21 MB 2025-02-14 10:03:22,038 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43444.60 MB 2025-02-14 10:03:22,038 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47219.47 MB 2025-02-14 10:03:22,038 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 10:03:22,038 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41691.30 MB 2025-02-14 10:03:22,210 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 10:03:22,210 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 10:03:22,210 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 10:03:22,210 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:03:22,210 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37680.56 MB 2025-02-14 10:03:22,210 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38447.56 MB 2025-02-14 10:03:22,210 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 10:03:22,210 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47219.47 MB 2025-02-14 10:03:22,210 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47634.71 MB 2025-02-14 10:03:22,210 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 10:03:22,210 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39155.35 MB 2025-02-14 10:03:22,229 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 10:03:22,229 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 10:03:22,229 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:03:22,229 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:03:22,229 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38860.45 MB 2025-02-14 10:03:22,229 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39088.92 MB 2025-02-14 10:03:22,229 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.47 MB 2025-02-14 10:03:22,229 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47634.71 MB 2025-02-14 10:03:22,229 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47634.71 MB 2025-02-14 10:03:22,229 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:03:22,229 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39326.19 MB 2025-02-14 10:03:22,230 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 10:03:22,231 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 10:03:22,231 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.01 seconds 2025-02-14 10:03:22,231 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:03:22,231 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26154.45 MB 2025-02-14 10:03:22,231 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39289.31 MB 2025-02-14 10:03:22,231 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13134.86 MB 2025-02-14 10:03:22,231 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48010.10 MB 2025-02-14 10:03:22,231 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47634.71 MB 2025-02-14 10:03:22,231 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -375.39 MB 2025-02-14 10:03:22,231 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39326.19 MB 2025-02-14 10:03:22,500 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 10:03:22,500 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 10:03:22,500 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 10:03:22,500 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:03:22,500 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39289.31 MB 2025-02-14 10:03:22,500 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31149.42 MB 2025-02-14 10:03:22,500 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8139.89 MB 2025-02-14 10:03:22,500 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47634.71 MB 2025-02-14 10:03:22,500 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47634.71 MB 2025-02-14 10:03:22,500 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:03:22,500 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41792.37 MB 2025-02-14 10:03:22,518 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8134, cut from 8136 2025-02-14 10:03:22,518 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 10:03:22,524 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 10:03:22,524 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 10:03:22,524 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:03:22,524 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:03:22,524 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31149.42 MB 2025-02-14 10:03:22,524 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39559.22 MB 2025-02-14 10:03:22,524 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8409.81 MB 2025-02-14 10:03:22,524 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47634.71 MB 2025-02-14 10:03:22,524 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55996.06 MB 2025-02-14 10:03:22,524 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8361.35 MB 2025-02-14 10:03:22,524 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39559.22 MB 2025-02-14 10:03:22,692 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7926] 2025-02-14 10:03:22,693 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:03:22,693 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 10:03:22,694 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:03:22,694 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 10:03:22,699 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 10:03:22,700 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:03:22,700 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 10:03:22,700 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 10:04:10,653 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:04:10,653 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 10:04:10,658 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 10:04:10,662 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:04:10,662 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 184, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 10:04:10,663 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:04:10,663 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 184, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 10:04:13,539 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 10:04:13,539 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 10:04:13,539 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.87 seconds 2025-02-14 10:04:13,539 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:04:13,539 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22314.99 MB 2025-02-14 10:04:13,539 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22966.16 MB 2025-02-14 10:04:13,539 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 651.17 MB 2025-02-14 10:04:13,539 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 68537.02 MB 2025-02-14 10:04:13,539 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26692.55 MB 2025-02-14 10:04:13,539 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -41844.47 MB 2025-02-14 10:04:13,539 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31787.44 MB 2025-02-14 10:04:13,552 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 10:04:13,552 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 10:04:13,552 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:04:13,552 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:04:13,552 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22966.16 MB 2025-02-14 10:04:13,552 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23162.65 MB 2025-02-14 10:04:13,552 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 196.49 MB 2025-02-14 10:04:13,552 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26692.55 MB 2025-02-14 10:04:13,552 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27283.95 MB 2025-02-14 10:04:13,552 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 591.40 MB 2025-02-14 10:04:13,552 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25340.62 MB 2025-02-14 10:04:14,397 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 10:04:14,397 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 10:04:14,397 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.84 seconds 2025-02-14 10:04:14,397 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:04:14,397 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23162.65 MB 2025-02-14 10:04:14,397 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23384.28 MB 2025-02-14 10:04:14,397 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 221.63 MB 2025-02-14 10:04:14,397 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27283.95 MB 2025-02-14 10:04:14,397 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27283.95 MB 2025-02-14 10:04:14,397 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:04:14,397 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27334.12 MB 2025-02-14 10:04:14,405 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 10:04:14,405 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 10:04:14,405 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 10:04:14,405 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:04:14,405 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23384.21 MB 2025-02-14 10:04:14,405 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24172.90 MB 2025-02-14 10:04:14,405 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 788.69 MB 2025-02-14 10:04:14,405 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27283.95 MB 2025-02-14 10:04:14,405 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27283.95 MB 2025-02-14 10:04:14,405 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:04:14,405 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24764.68 MB 2025-02-14 10:04:14,495 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 10:04:14,495 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 10:04:14,495 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 10:04:14,495 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:04:14,495 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24172.90 MB 2025-02-14 10:04:14,495 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25108.91 MB 2025-02-14 10:04:14,495 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 936.01 MB 2025-02-14 10:04:14,495 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27283.95 MB 2025-02-14 10:04:14,495 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28861.01 MB 2025-02-14 10:04:14,495 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1577.06 MB 2025-02-14 10:04:14,495 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27424.40 MB 2025-02-14 10:04:14,496 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 10:04:14,496 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 10:04:14,496 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 10:04:14,496 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:04:14,496 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23384.21 MB 2025-02-14 10:04:14,496 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25108.91 MB 2025-02-14 10:04:14,496 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1724.70 MB 2025-02-14 10:04:14,496 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27283.95 MB 2025-02-14 10:04:14,496 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28861.01 MB 2025-02-14 10:04:14,496 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1577.06 MB 2025-02-14 10:04:14,496 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27424.40 MB 2025-02-14 10:04:14,567 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 10:04:14,567 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 10:04:14,567 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 10:04:14,567 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:04:14,567 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25749.17 MB 2025-02-14 10:04:14,567 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26069.39 MB 2025-02-14 10:04:14,567 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 320.22 MB 2025-02-14 10:04:14,567 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28861.01 MB 2025-02-14 10:04:14,567 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29032.97 MB 2025-02-14 10:04:14,567 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 171.97 MB 2025-02-14 10:04:14,567 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26372.88 MB 2025-02-14 10:04:14,577 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 10:04:14,577 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 10:04:14,577 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:04:14,577 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:04:14,577 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26241.78 MB 2025-02-14 10:04:14,577 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26465.77 MB 2025-02-14 10:04:14,577 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 223.99 MB 2025-02-14 10:04:14,577 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29032.97 MB 2025-02-14 10:04:14,577 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29032.97 MB 2025-02-14 10:04:14,577 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:04:14,577 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26485.50 MB 2025-02-14 10:04:14,578 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 10:04:14,578 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 10:04:14,578 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.91 seconds 2025-02-14 10:04:14,578 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:04:14,578 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21673.92 MB 2025-02-14 10:04:14,578 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26666.77 MB 2025-02-14 10:04:14,578 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4992.85 MB 2025-02-14 10:04:14,578 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 68537.02 MB 2025-02-14 10:04:14,578 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29032.97 MB 2025-02-14 10:04:14,578 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -39504.05 MB 2025-02-14 10:04:14,578 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26666.77 MB 2025-02-14 10:04:14,844 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 10:04:14,844 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 10:04:14,844 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 10:04:14,844 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:04:14,844 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26666.77 MB 2025-02-14 10:04:14,844 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25577.43 MB 2025-02-14 10:04:14,844 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1089.34 MB 2025-02-14 10:04:14,844 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29032.97 MB 2025-02-14 10:04:14,844 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29032.97 MB 2025-02-14 10:04:14,844 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:04:14,844 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27269.35 MB 2025-02-14 10:04:14,862 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8159, cut from 8161 2025-02-14 10:04:14,862 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 10:04:14,869 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 10:04:14,869 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 10:04:14,869 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:04:14,869 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:04:14,869 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25577.43 MB 2025-02-14 10:04:14,869 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34013.02 MB 2025-02-14 10:04:14,869 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8435.59 MB 2025-02-14 10:04:14,869 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29032.97 MB 2025-02-14 10:04:14,869 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39518.73 MB 2025-02-14 10:04:14,869 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10485.76 MB 2025-02-14 10:04:14,869 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34013.02 MB 2025-02-14 10:04:15,034 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7951] 2025-02-14 10:04:15,035 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:04:15,035 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 10:04:15,036 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:04:15,036 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 10:04:15,041 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 10:04:15,042 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:04:15,042 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 10:04:15,042 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 10:06:00,985 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:06:00,986 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 10:06:00,990 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 10:06:00,995 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:06:00,995 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1133, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 10:06:00,996 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:06:00,996 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1133, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 10:06:18,288 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 10:06:18,288 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 10:06:18,288 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.28 seconds 2025-02-14 10:06:18,288 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:06:18,288 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28927.78 MB 2025-02-14 10:06:18,288 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32937.53 MB 2025-02-14 10:06:18,288 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4009.75 MB 2025-02-14 10:06:18,288 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47907.34 MB 2025-02-14 10:06:18,288 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41188.07 MB 2025-02-14 10:06:18,288 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6719.28 MB 2025-02-14 10:06:18,288 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41796.53 MB 2025-02-14 10:06:18,350 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 10:06:18,351 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 10:06:18,351 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 10:06:18,351 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:06:18,351 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32937.53 MB 2025-02-14 10:06:18,351 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29732.11 MB 2025-02-14 10:06:18,351 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3205.42 MB 2025-02-14 10:06:18,351 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41188.07 MB 2025-02-14 10:06:18,351 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48052.04 MB 2025-02-14 10:06:18,351 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6863.98 MB 2025-02-14 10:06:18,351 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44070.93 MB 2025-02-14 10:06:20,244 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 10:06:20,244 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 10:06:20,244 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.89 seconds 2025-02-14 10:06:20,244 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:06:20,244 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29732.11 MB 2025-02-14 10:06:20,244 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30262.95 MB 2025-02-14 10:06:20,244 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 10:06:20,244 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48052.04 MB 2025-02-14 10:06:20,244 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37178.31 MB 2025-02-14 10:06:20,245 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10873.73 MB 2025-02-14 10:06:20,245 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34242.28 MB 2025-02-14 10:06:20,258 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 10:06:20,258 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 10:06:20,258 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:06:20,258 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:06:20,258 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30262.95 MB 2025-02-14 10:06:20,258 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32152.30 MB 2025-02-14 10:06:20,258 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.35 MB 2025-02-14 10:06:20,258 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37178.31 MB 2025-02-14 10:06:20,258 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37178.31 MB 2025-02-14 10:06:20,258 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:06:20,258 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33569.73 MB 2025-02-14 10:06:20,467 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 10:06:20,467 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 10:06:20,467 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 10:06:20,467 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:06:20,467 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32152.30 MB 2025-02-14 10:06:20,467 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34394.16 MB 2025-02-14 10:06:20,467 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 10:06:20,467 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37178.31 MB 2025-02-14 10:06:20,467 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42840.62 MB 2025-02-14 10:06:20,467 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 10:06:20,467 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39939.34 MB 2025-02-14 10:06:20,468 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 10:06:20,468 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 10:06:20,468 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 10:06:20,468 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:06:20,468 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30262.95 MB 2025-02-14 10:06:20,468 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34394.16 MB 2025-02-14 10:06:20,468 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.21 MB 2025-02-14 10:06:20,468 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37178.31 MB 2025-02-14 10:06:20,468 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42840.62 MB 2025-02-14 10:06:20,468 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 10:06:20,468 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39939.34 MB 2025-02-14 10:06:20,640 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 10:06:20,641 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 10:06:20,641 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 10:06:20,641 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:06:20,641 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35928.60 MB 2025-02-14 10:06:20,641 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36695.60 MB 2025-02-14 10:06:20,641 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 10:06:20,641 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42840.62 MB 2025-02-14 10:06:20,641 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43255.86 MB 2025-02-14 10:06:20,641 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 10:06:20,641 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37403.39 MB 2025-02-14 10:06:20,660 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 10:06:20,660 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 10:06:20,660 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:06:20,660 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:06:20,660 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37108.49 MB 2025-02-14 10:06:20,660 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37336.23 MB 2025-02-14 10:06:20,660 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.73 MB 2025-02-14 10:06:20,660 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43255.86 MB 2025-02-14 10:06:20,660 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43255.86 MB 2025-02-14 10:06:20,660 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:06:20,660 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37521.42 MB 2025-02-14 10:06:20,661 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 10:06:20,661 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 10:06:20,661 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.66 seconds 2025-02-14 10:06:20,661 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:06:20,661 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24980.31 MB 2025-02-14 10:06:20,661 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37536.56 MB 2025-02-14 10:06:20,661 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12556.25 MB 2025-02-14 10:06:20,661 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47907.34 MB 2025-02-14 10:06:20,661 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43255.86 MB 2025-02-14 10:06:20,661 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4651.48 MB 2025-02-14 10:06:20,661 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37536.56 MB 2025-02-14 10:06:20,927 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 10:06:20,927 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 10:06:20,927 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 10:06:20,927 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:06:20,927 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37536.56 MB 2025-02-14 10:06:20,928 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29974.54 MB 2025-02-14 10:06:20,928 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7562.02 MB 2025-02-14 10:06:20,928 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43255.86 MB 2025-02-14 10:06:20,928 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43255.86 MB 2025-02-14 10:06:20,928 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:06:20,928 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40039.01 MB 2025-02-14 10:06:20,945 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8132, cut from 8134 2025-02-14 10:06:20,946 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 10:06:20,952 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 10:06:20,952 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 10:06:20,952 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:06:20,952 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:06:20,952 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29974.54 MB 2025-02-14 10:06:20,952 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38383.84 MB 2025-02-14 10:06:20,952 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8409.30 MB 2025-02-14 10:06:20,952 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43255.86 MB 2025-02-14 10:06:20,952 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51615.11 MB 2025-02-14 10:06:20,952 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8359.25 MB 2025-02-14 10:06:20,952 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38383.84 MB 2025-02-14 10:06:21,112 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7924] 2025-02-14 10:06:21,113 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:06:21,113 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 10:06:21,114 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:06:21,114 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 10:06:21,119 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 10:06:21,120 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:06:21,120 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 10:06:21,120 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 10:06:42,633 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:06:42,633 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 10:06:42,638 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 10:06:42,642 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:06:42,642 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2375, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 10:06:42,643 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:06:42,643 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2375, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 10:07:19,482 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 10:07:19,482 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 10:07:19,482 - resource_logging.py:150 - __exit__ - DEBUG - Time: 36.83 seconds 2025-02-14 10:07:19,482 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:07:19,482 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37582.62 MB 2025-02-14 10:07:19,482 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45988.01 MB 2025-02-14 10:07:19,482 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8405.39 MB 2025-02-14 10:07:19,482 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 68379.74 MB 2025-02-14 10:07:19,482 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51866.76 MB 2025-02-14 10:07:19,482 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16512.97 MB 2025-02-14 10:07:19,482 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54982.04 MB 2025-02-14 10:07:19,628 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 10:07:19,628 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 10:07:19,628 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-14 10:07:19,628 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:07:19,629 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45988.01 MB 2025-02-14 10:07:19,629 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36190.13 MB 2025-02-14 10:07:19,629 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9797.88 MB 2025-02-14 10:07:19,629 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51866.76 MB 2025-02-14 10:07:19,629 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 79981.18 MB 2025-02-14 10:07:19,629 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 28114.42 MB 2025-02-14 10:07:19,629 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 69632.56 MB 2025-02-14 10:07:21,589 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 10:07:21,589 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 10:07:21,589 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.96 seconds 2025-02-14 10:07:21,589 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:07:21,589 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36190.13 MB 2025-02-14 10:07:21,589 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36720.97 MB 2025-02-14 10:07:21,589 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 10:07:21,589 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 79981.18 MB 2025-02-14 10:07:21,589 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41383.10 MB 2025-02-14 10:07:21,589 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -38598.08 MB 2025-02-14 10:07:21,589 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40701.34 MB 2025-02-14 10:07:21,603 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 10:07:21,603 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 10:07:21,603 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:07:21,603 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:07:21,603 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36720.97 MB 2025-02-14 10:07:21,603 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38610.32 MB 2025-02-14 10:07:21,603 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.35 MB 2025-02-14 10:07:21,603 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41383.10 MB 2025-02-14 10:07:21,603 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43270.54 MB 2025-02-14 10:07:21,603 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 10:07:21,603 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40027.75 MB 2025-02-14 10:07:21,814 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 10:07:21,814 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 10:07:21,814 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 10:07:21,814 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:07:21,814 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38610.32 MB 2025-02-14 10:07:21,814 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40852.18 MB 2025-02-14 10:07:21,814 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 10:07:21,814 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43270.54 MB 2025-02-14 10:07:21,814 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49404.71 MB 2025-02-14 10:07:21,814 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 10:07:21,814 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46397.36 MB 2025-02-14 10:07:21,815 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 10:07:21,815 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 10:07:21,815 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 10:07:21,815 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:07:21,815 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36720.97 MB 2025-02-14 10:07:21,815 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40852.18 MB 2025-02-14 10:07:21,815 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.21 MB 2025-02-14 10:07:21,815 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41383.10 MB 2025-02-14 10:07:21,815 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49404.71 MB 2025-02-14 10:07:21,815 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-14 10:07:21,815 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46397.36 MB 2025-02-14 10:07:21,982 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 10:07:21,982 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 10:07:21,982 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 10:07:21,982 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:07:21,982 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42386.62 MB 2025-02-14 10:07:21,982 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43153.62 MB 2025-02-14 10:07:21,982 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 10:07:21,982 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49404.71 MB 2025-02-14 10:07:21,982 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49815.75 MB 2025-02-14 10:07:21,982 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 411.04 MB 2025-02-14 10:07:21,982 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43861.41 MB 2025-02-14 10:07:22,001 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 10:07:22,001 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 10:07:22,001 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:07:22,001 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:07:22,001 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43566.51 MB 2025-02-14 10:07:22,001 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43795.32 MB 2025-02-14 10:07:22,001 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.81 MB 2025-02-14 10:07:22,001 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49815.75 MB 2025-02-14 10:07:22,001 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49815.75 MB 2025-02-14 10:07:22,001 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:07:22,001 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44008.93 MB 2025-02-14 10:07:22,002 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 10:07:22,002 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 10:07:22,002 - resource_logging.py:150 - __exit__ - DEBUG - Time: 39.36 seconds 2025-02-14 10:07:22,002 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:07:22,002 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29307.74 MB 2025-02-14 10:07:22,002 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43996.05 MB 2025-02-14 10:07:22,002 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14688.31 MB 2025-02-14 10:07:22,002 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64177.05 MB 2025-02-14 10:07:22,002 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49815.75 MB 2025-02-14 10:07:22,002 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14361.30 MB 2025-02-14 10:07:22,002 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44008.93 MB 2025-02-14 10:07:22,272 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 10:07:22,272 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 10:07:22,272 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 10:07:22,273 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:07:22,273 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43996.05 MB 2025-02-14 10:07:22,273 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34307.87 MB 2025-02-14 10:07:22,273 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9688.18 MB 2025-02-14 10:07:22,273 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49815.75 MB 2025-02-14 10:07:22,273 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49815.75 MB 2025-02-14 10:07:22,273 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:07:22,273 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46503.42 MB 2025-02-14 10:07:22,290 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8148, cut from 8150 2025-02-14 10:07:22,291 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 10:07:22,297 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 10:07:22,297 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 10:07:22,297 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:07:22,297 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:07:22,297 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34307.87 MB 2025-02-14 10:07:22,297 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42732.82 MB 2025-02-14 10:07:22,297 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8424.95 MB 2025-02-14 10:07:22,297 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49815.75 MB 2025-02-14 10:07:22,297 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58191.77 MB 2025-02-14 10:07:22,297 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8376.03 MB 2025-02-14 10:07:22,297 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42732.82 MB 2025-02-14 10:07:22,462 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7940] 2025-02-14 10:07:22,463 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:07:22,463 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 10:07:22,465 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:07:22,465 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 10:07:22,470 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 10:07:22,471 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:07:22,471 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 10:07:22,471 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 10:07:44,272 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:07:44,272 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 10:07:44,277 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 10:07:44,281 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:07:44,281 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 386, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 10:07:44,282 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:07:44,282 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 386, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 10:07:50,313 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 10:07:50,313 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 10:07:50,313 - resource_logging.py:150 - __exit__ - DEBUG - Time: 6.03 seconds 2025-02-14 10:07:50,313 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:07:50,313 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23722.56 MB 2025-02-14 10:07:50,313 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25088.59 MB 2025-02-14 10:07:50,313 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1366.03 MB 2025-02-14 10:07:50,313 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 66567.80 MB 2025-02-14 10:07:50,313 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28347.20 MB 2025-02-14 10:07:50,313 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -38220.60 MB 2025-02-14 10:07:50,313 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34101.23 MB 2025-02-14 10:07:50,336 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 10:07:50,336 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 10:07:50,336 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:07:50,336 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:07:50,336 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25088.59 MB 2025-02-14 10:07:50,336 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25119.87 MB 2025-02-14 10:07:50,336 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 31.27 MB 2025-02-14 10:07:50,336 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28347.20 MB 2025-02-14 10:07:50,336 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31987.86 MB 2025-02-14 10:07:50,336 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3640.66 MB 2025-02-14 10:07:50,336 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29255.92 MB 2025-02-14 10:07:51,752 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 10:07:51,752 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 10:07:51,752 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.41 seconds 2025-02-14 10:07:51,752 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:07:51,752 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25119.87 MB 2025-02-14 10:07:51,752 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25512.69 MB 2025-02-14 10:07:51,752 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 392.82 MB 2025-02-14 10:07:51,752 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31987.86 MB 2025-02-14 10:07:51,752 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30429.68 MB 2025-02-14 10:07:51,752 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1558.18 MB 2025-02-14 10:07:51,752 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29460.17 MB 2025-02-14 10:07:51,763 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 10:07:51,763 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 10:07:51,763 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:07:51,763 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:07:51,763 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25512.69 MB 2025-02-14 10:07:51,763 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26911.31 MB 2025-02-14 10:07:51,763 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1398.62 MB 2025-02-14 10:07:51,763 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30429.68 MB 2025-02-14 10:07:51,763 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30429.68 MB 2025-02-14 10:07:51,763 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:07:51,763 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27960.21 MB 2025-02-14 10:07:51,919 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 10:07:51,919 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 10:07:51,919 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 10:07:51,919 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:07:51,919 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26911.31 MB 2025-02-14 10:07:51,919 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28570.30 MB 2025-02-14 10:07:51,919 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1658.99 MB 2025-02-14 10:07:51,919 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30429.68 MB 2025-02-14 10:07:51,919 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34619.79 MB 2025-02-14 10:07:51,919 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4190.11 MB 2025-02-14 10:07:51,919 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32673.05 MB 2025-02-14 10:07:51,920 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 10:07:51,920 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 10:07:51,920 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 10:07:51,920 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:07:51,920 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25512.69 MB 2025-02-14 10:07:51,920 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28570.30 MB 2025-02-14 10:07:51,920 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3057.61 MB 2025-02-14 10:07:51,920 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30429.68 MB 2025-02-14 10:07:51,920 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34619.79 MB 2025-02-14 10:07:51,920 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4190.11 MB 2025-02-14 10:07:51,920 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32673.05 MB 2025-02-14 10:07:52,045 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 10:07:52,045 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 10:07:52,045 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 10:07:52,045 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:07:52,045 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29705.12 MB 2025-02-14 10:07:52,045 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30272.70 MB 2025-02-14 10:07:52,045 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 567.58 MB 2025-02-14 10:07:52,045 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34619.79 MB 2025-02-14 10:07:52,045 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34923.87 MB 2025-02-14 10:07:52,045 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 304.09 MB 2025-02-14 10:07:52,045 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30796.47 MB 2025-02-14 10:07:52,060 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 10:07:52,060 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 10:07:52,060 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:07:52,060 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:07:52,060 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30578.25 MB 2025-02-14 10:07:52,060 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30806.47 MB 2025-02-14 10:07:52,060 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.22 MB 2025-02-14 10:07:52,060 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34923.87 MB 2025-02-14 10:07:52,060 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34923.87 MB 2025-02-14 10:07:52,060 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:07:52,060 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30900.54 MB 2025-02-14 10:07:52,061 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 10:07:52,061 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 10:07:52,061 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.78 seconds 2025-02-14 10:07:52,061 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:07:52,061 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22377.71 MB 2025-02-14 10:07:52,061 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31007.54 MB 2025-02-14 10:07:52,061 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8629.83 MB 2025-02-14 10:07:52,061 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 66567.80 MB 2025-02-14 10:07:52,061 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34923.87 MB 2025-02-14 10:07:52,061 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31643.93 MB 2025-02-14 10:07:52,061 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31007.54 MB 2025-02-14 10:07:52,332 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 10:07:52,332 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 10:07:52,332 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 10:07:52,332 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:07:52,332 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31007.54 MB 2025-02-14 10:07:52,332 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34021.57 MB 2025-02-14 10:07:52,332 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-14 10:07:52,332 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34923.87 MB 2025-02-14 10:07:52,332 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35863.40 MB 2025-02-14 10:07:52,332 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 939.52 MB 2025-02-14 10:07:52,332 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34323.20 MB 2025-02-14 10:07:52,350 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 10:07:52,350 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-14 10:07:52,356 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 10:07:52,356 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 10:07:52,356 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:07:52,356 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:07:52,356 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26891.28 MB 2025-02-14 10:07:52,356 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35330.30 MB 2025-02-14 10:07:52,356 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 10:07:52,356 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35863.40 MB 2025-02-14 10:07:52,356 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46353.35 MB 2025-02-14 10:07:52,356 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 10:07:52,356 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35330.30 MB 2025-02-14 10:07:52,518 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 10:07:52,519 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:07:52,519 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 10:07:52,520 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:07:52,520 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 10:07:52,525 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 10:07:52,526 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:07:52,526 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 10:07:52,526 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-14 10:08:05,920 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:08:05,920 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 10:08:05,925 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 10:08:05,928 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:08:05,928 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 414, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 10:08:05,929 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:08:05,929 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 414, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 10:08:12,383 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 10:08:12,383 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 10:08:12,383 - resource_logging.py:150 - __exit__ - DEBUG - Time: 6.45 seconds 2025-02-14 10:08:12,383 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:08:12,383 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23917.67 MB 2025-02-14 10:08:12,383 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25382.79 MB 2025-02-14 10:08:12,383 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1465.12 MB 2025-02-14 10:08:12,383 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58938.36 MB 2025-02-14 10:08:12,383 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28684.85 MB 2025-02-14 10:08:12,383 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30253.51 MB 2025-02-14 10:08:12,383 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34296.39 MB 2025-02-14 10:08:12,410 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 10:08:12,410 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 10:08:12,410 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 10:08:12,410 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:08:12,410 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25382.79 MB 2025-02-14 10:08:12,410 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25685.76 MB 2025-02-14 10:08:12,410 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 302.97 MB 2025-02-14 10:08:12,410 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28684.85 MB 2025-02-14 10:08:12,410 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33692.84 MB 2025-02-14 10:08:12,410 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5008.00 MB 2025-02-14 10:08:12,410 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30433.78 MB 2025-02-14 10:08:14,125 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 10:08:14,126 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 10:08:14,126 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.71 seconds 2025-02-14 10:08:14,126 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:08:14,126 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25685.76 MB 2025-02-14 10:08:14,126 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26158.21 MB 2025-02-14 10:08:14,126 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 472.45 MB 2025-02-14 10:08:14,126 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33692.84 MB 2025-02-14 10:08:14,126 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31188.84 MB 2025-02-14 10:08:14,126 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2504.00 MB 2025-02-14 10:08:14,126 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30111.01 MB 2025-02-14 10:08:14,141 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 10:08:14,141 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 10:08:14,141 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:08:14,141 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:08:14,141 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26158.21 MB 2025-02-14 10:08:14,141 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27842.05 MB 2025-02-14 10:08:14,141 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1683.83 MB 2025-02-14 10:08:14,141 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31188.84 MB 2025-02-14 10:08:14,141 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32870.76 MB 2025-02-14 10:08:14,141 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1681.92 MB 2025-02-14 10:08:14,141 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29103.56 MB 2025-02-14 10:08:14,330 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 10:08:14,330 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 10:08:14,330 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.19 seconds 2025-02-14 10:08:14,330 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:08:14,330 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27842.05 MB 2025-02-14 10:08:14,330 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29838.35 MB 2025-02-14 10:08:14,330 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1996.31 MB 2025-02-14 10:08:14,330 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32870.76 MB 2025-02-14 10:08:14,330 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37916.51 MB 2025-02-14 10:08:14,330 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5045.75 MB 2025-02-14 10:08:14,330 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34776.95 MB 2025-02-14 10:08:14,331 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 10:08:14,331 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 10:08:14,331 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 10:08:14,331 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:08:14,331 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26158.21 MB 2025-02-14 10:08:14,331 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29838.35 MB 2025-02-14 10:08:14,331 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3680.14 MB 2025-02-14 10:08:14,331 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31188.84 MB 2025-02-14 10:08:14,331 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37916.51 MB 2025-02-14 10:08:14,331 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6727.66 MB 2025-02-14 10:08:14,331 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34776.95 MB 2025-02-14 10:08:14,481 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 10:08:14,481 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 10:08:14,481 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-14 10:08:14,481 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:08:14,481 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31203.21 MB 2025-02-14 10:08:14,481 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31885.84 MB 2025-02-14 10:08:14,481 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 682.63 MB 2025-02-14 10:08:14,481 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37916.51 MB 2025-02-14 10:08:14,481 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38285.61 MB 2025-02-14 10:08:14,481 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 369.10 MB 2025-02-14 10:08:14,481 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32515.77 MB 2025-02-14 10:08:14,498 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 10:08:14,498 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 10:08:14,498 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:08:14,498 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:08:14,498 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32253.31 MB 2025-02-14 10:08:14,498 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32467.89 MB 2025-02-14 10:08:14,498 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 214.58 MB 2025-02-14 10:08:14,498 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38285.61 MB 2025-02-14 10:08:14,498 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38287.70 MB 2025-02-14 10:08:14,498 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-14 10:08:14,498 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32617.92 MB 2025-02-14 10:08:14,499 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 10:08:14,499 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 10:08:14,499 - resource_logging.py:150 - __exit__ - DEBUG - Time: 8.57 seconds 2025-02-14 10:08:14,499 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:08:14,499 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22475.26 MB 2025-02-14 10:08:14,499 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32668.96 MB 2025-02-14 10:08:14,499 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 10193.70 MB 2025-02-14 10:08:14,499 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58938.36 MB 2025-02-14 10:08:14,499 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38287.70 MB 2025-02-14 10:08:14,499 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20650.66 MB 2025-02-14 10:08:14,500 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32668.96 MB 2025-02-14 10:08:14,766 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 10:08:14,766 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 10:08:14,766 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 10:08:14,766 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:08:14,766 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32668.96 MB 2025-02-14 10:08:14,767 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27271.87 MB 2025-02-14 10:08:14,767 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5397.09 MB 2025-02-14 10:08:14,767 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38287.70 MB 2025-02-14 10:08:14,767 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38287.70 MB 2025-02-14 10:08:14,767 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:08:14,767 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35883.90 MB 2025-02-14 10:08:14,785 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 10:08:14,785 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 10:08:14,791 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 10:08:14,791 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 10:08:14,791 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:08:14,791 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:08:14,791 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27271.87 MB 2025-02-14 10:08:14,791 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35710.90 MB 2025-02-14 10:08:14,791 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 10:08:14,791 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38287.70 MB 2025-02-14 10:08:14,791 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48777.66 MB 2025-02-14 10:08:14,791 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 10:08:14,791 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35710.90 MB 2025-02-14 10:08:14,954 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 10:08:14,955 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:08:14,955 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 10:08:14,956 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:08:14,956 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 10:08:14,961 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 10:08:14,962 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:08:14,962 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 10:08:14,962 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 10:09:00,708 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:09:00,708 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 10:09:00,713 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 10:09:00,716 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:09:00,717 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 283, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 10:09:00,717 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:09:00,717 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 283, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 10:09:05,095 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 10:09:05,095 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 10:09:05,095 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.37 seconds 2025-02-14 10:09:05,095 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:09:05,095 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23004.84 MB 2025-02-14 10:09:05,095 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24006.36 MB 2025-02-14 10:09:05,095 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1001.52 MB 2025-02-14 10:09:05,095 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61362.67 MB 2025-02-14 10:09:05,095 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27938.26 MB 2025-02-14 10:09:05,095 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33424.41 MB 2025-02-14 10:09:05,095 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32930.00 MB 2025-02-14 10:09:05,113 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 10:09:05,113 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 10:09:05,113 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:09:05,113 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:09:05,113 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24006.36 MB 2025-02-14 10:09:05,113 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24006.94 MB 2025-02-14 10:09:05,113 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 0.58 MB 2025-02-14 10:09:05,113 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27938.26 MB 2025-02-14 10:09:05,113 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28697.43 MB 2025-02-14 10:09:05,113 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 759.17 MB 2025-02-14 10:09:05,113 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27051.14 MB 2025-02-14 10:09:06,134 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 10:09:06,134 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 10:09:06,134 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.02 seconds 2025-02-14 10:09:06,134 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:09:06,135 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24006.94 MB 2025-02-14 10:09:06,135 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24290.94 MB 2025-02-14 10:09:06,135 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 284.00 MB 2025-02-14 10:09:06,135 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28697.43 MB 2025-02-14 10:09:06,135 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29100.08 MB 2025-02-14 10:09:06,135 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 402.65 MB 2025-02-14 10:09:06,135 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28263.35 MB 2025-02-14 10:09:06,143 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 10:09:06,143 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 10:09:06,143 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:09:06,143 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:09:06,143 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24290.94 MB 2025-02-14 10:09:06,143 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25301.60 MB 2025-02-14 10:09:06,143 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1010.66 MB 2025-02-14 10:09:06,143 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29100.08 MB 2025-02-14 10:09:06,143 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29100.08 MB 2025-02-14 10:09:06,143 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:09:06,143 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26059.93 MB 2025-02-14 10:09:06,260 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 10:09:06,260 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 10:09:06,260 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 10:09:06,260 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:09:06,260 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25301.60 MB 2025-02-14 10:09:06,260 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26501.93 MB 2025-02-14 10:09:06,260 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1200.33 MB 2025-02-14 10:09:06,260 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29100.08 MB 2025-02-14 10:09:06,260 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31627.15 MB 2025-02-14 10:09:06,260 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2527.07 MB 2025-02-14 10:09:06,260 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29470.19 MB 2025-02-14 10:09:06,261 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 10:09:06,261 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 10:09:06,261 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 10:09:06,261 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:09:06,261 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24290.94 MB 2025-02-14 10:09:06,261 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26501.93 MB 2025-02-14 10:09:06,261 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2210.99 MB 2025-02-14 10:09:06,261 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29100.08 MB 2025-02-14 10:09:06,261 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31627.15 MB 2025-02-14 10:09:06,261 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2527.07 MB 2025-02-14 10:09:06,261 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29470.19 MB 2025-02-14 10:09:06,351 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 10:09:06,351 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 10:09:06,351 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 10:09:06,351 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:09:06,351 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27322.38 MB 2025-02-14 10:09:06,351 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27732.72 MB 2025-02-14 10:09:06,351 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 410.35 MB 2025-02-14 10:09:06,351 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31627.15 MB 2025-02-14 10:09:06,351 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31847.35 MB 2025-02-14 10:09:06,351 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 220.20 MB 2025-02-14 10:09:06,351 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28112.81 MB 2025-02-14 10:09:06,363 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 10:09:06,363 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 10:09:06,363 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:09:06,363 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:09:06,363 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27953.63 MB 2025-02-14 10:09:06,363 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28174.48 MB 2025-02-14 10:09:06,363 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 220.86 MB 2025-02-14 10:09:06,363 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31847.35 MB 2025-02-14 10:09:06,363 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31847.35 MB 2025-02-14 10:09:06,363 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:09:06,363 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28207.79 MB 2025-02-14 10:09:06,364 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 10:09:06,364 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 10:09:06,364 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.64 seconds 2025-02-14 10:09:06,364 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:09:06,364 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22018.85 MB 2025-02-14 10:09:06,364 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28375.24 MB 2025-02-14 10:09:06,364 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6356.39 MB 2025-02-14 10:09:06,364 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61362.67 MB 2025-02-14 10:09:06,364 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31847.35 MB 2025-02-14 10:09:06,364 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29515.32 MB 2025-02-14 10:09:06,364 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28375.24 MB 2025-02-14 10:09:06,630 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 10:09:06,631 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 10:09:06,631 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 10:09:06,631 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:09:06,631 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23130.99 MB 2025-02-14 10:09:06,631 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26141.03 MB 2025-02-14 10:09:06,631 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3010.04 MB 2025-02-14 10:09:06,631 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31847.35 MB 2025-02-14 10:09:06,631 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31847.35 MB 2025-02-14 10:09:06,631 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:09:06,631 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26441.92 MB 2025-02-14 10:09:06,648 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8149, cut from 8151 2025-02-14 10:09:06,649 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 10:09:06,655 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 10:09:06,655 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 10:09:06,655 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:09:06,655 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:09:06,655 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26141.03 MB 2025-02-14 10:09:06,655 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34567.21 MB 2025-02-14 10:09:06,655 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8426.18 MB 2025-02-14 10:09:06,655 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31847.35 MB 2025-02-14 10:09:06,655 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42318.43 MB 2025-02-14 10:09:06,655 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10471.08 MB 2025-02-14 10:09:06,655 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34567.21 MB 2025-02-14 10:09:06,819 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7941] 2025-02-14 10:09:06,820 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:09:06,820 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 10:09:06,821 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:09:06,821 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 10:09:06,826 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 10:09:06,827 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:09:06,827 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 10:09:06,827 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 10:09:20,094 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:09:20,094 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 10:09:20,098 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 10:09:20,102 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:09:20,102 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1025, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 10:09:20,103 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:09:20,103 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1025, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 10:09:35,954 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 10:09:35,954 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 10:09:35,954 - resource_logging.py:150 - __exit__ - DEBUG - Time: 15.85 seconds 2025-02-14 10:09:35,954 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:09:35,954 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28175.22 MB 2025-02-14 10:09:35,954 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31803.29 MB 2025-02-14 10:09:35,955 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3628.07 MB 2025-02-14 10:09:35,955 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50694.46 MB 2025-02-14 10:09:35,955 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40791.70 MB 2025-02-14 10:09:35,955 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9902.75 MB 2025-02-14 10:09:35,955 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40817.48 MB 2025-02-14 10:09:36,015 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 10:09:36,015 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 10:09:36,015 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 10:09:36,015 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:09:36,015 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31803.29 MB 2025-02-14 10:09:36,015 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29170.65 MB 2025-02-14 10:09:36,015 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2632.64 MB 2025-02-14 10:09:36,015 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40791.70 MB 2025-02-14 10:09:36,016 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47942.99 MB 2025-02-14 10:09:36,016 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7151.29 MB 2025-02-14 10:09:36,016 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43181.45 MB 2025-02-14 10:09:37,933 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 10:09:37,933 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 10:09:37,933 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 10:09:37,933 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:09:37,933 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29170.65 MB 2025-02-14 10:09:37,933 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29701.49 MB 2025-02-14 10:09:37,933 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 10:09:37,933 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47942.99 MB 2025-02-14 10:09:37,933 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37163.63 MB 2025-02-14 10:09:37,933 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10779.36 MB 2025-02-14 10:09:37,933 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33680.83 MB 2025-02-14 10:09:37,946 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 10:09:37,946 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 10:09:37,946 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:09:37,946 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:09:37,946 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29701.49 MB 2025-02-14 10:09:37,946 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31590.85 MB 2025-02-14 10:09:37,946 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.35 MB 2025-02-14 10:09:37,946 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37163.63 MB 2025-02-14 10:09:37,946 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37163.63 MB 2025-02-14 10:09:37,946 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:09:37,946 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33008.28 MB 2025-02-14 10:09:38,161 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 10:09:38,161 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 10:09:38,161 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 10:09:38,161 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:09:38,161 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31590.85 MB 2025-02-14 10:09:38,161 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33832.70 MB 2025-02-14 10:09:38,161 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 10:09:38,161 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37163.63 MB 2025-02-14 10:09:38,161 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42825.94 MB 2025-02-14 10:09:38,161 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 10:09:38,161 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39377.88 MB 2025-02-14 10:09:38,162 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 10:09:38,162 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 10:09:38,162 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 10:09:38,162 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:09:38,162 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29701.49 MB 2025-02-14 10:09:38,162 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33832.70 MB 2025-02-14 10:09:38,162 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.21 MB 2025-02-14 10:09:38,162 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37163.63 MB 2025-02-14 10:09:38,162 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42825.94 MB 2025-02-14 10:09:38,162 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 10:09:38,162 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39377.88 MB 2025-02-14 10:09:38,365 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 10:09:38,365 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 10:09:38,365 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 10:09:38,365 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:09:38,365 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35367.14 MB 2025-02-14 10:09:38,365 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36134.15 MB 2025-02-14 10:09:38,365 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 10:09:38,365 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42825.94 MB 2025-02-14 10:09:38,365 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43241.18 MB 2025-02-14 10:09:38,365 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 10:09:38,365 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36841.94 MB 2025-02-14 10:09:38,389 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 10:09:38,389 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 10:09:38,389 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:09:38,389 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:09:38,389 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36547.04 MB 2025-02-14 10:09:38,389 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36774.42 MB 2025-02-14 10:09:38,389 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.39 MB 2025-02-14 10:09:38,389 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43241.18 MB 2025-02-14 10:09:38,389 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43241.18 MB 2025-02-14 10:09:38,389 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:09:38,389 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37007.11 MB 2025-02-14 10:09:38,390 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 10:09:38,390 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 10:09:38,390 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.29 seconds 2025-02-14 10:09:38,390 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:09:38,390 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24604.03 MB 2025-02-14 10:09:38,390 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36974.88 MB 2025-02-14 10:09:38,390 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12370.85 MB 2025-02-14 10:09:38,390 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50694.46 MB 2025-02-14 10:09:38,390 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43241.18 MB 2025-02-14 10:09:38,390 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7453.28 MB 2025-02-14 10:09:38,390 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37007.11 MB 2025-02-14 10:09:38,659 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 10:09:38,659 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 10:09:38,659 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 10:09:38,659 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:09:38,659 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36974.88 MB 2025-02-14 10:09:38,659 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29600.11 MB 2025-02-14 10:09:38,659 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7374.77 MB 2025-02-14 10:09:38,659 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43241.18 MB 2025-02-14 10:09:38,659 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43241.18 MB 2025-02-14 10:09:38,659 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:09:38,659 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39478.87 MB 2025-02-14 10:09:38,677 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8137, cut from 8139 2025-02-14 10:09:38,677 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 10:09:38,683 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 10:09:38,683 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 10:09:38,683 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:09:38,683 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:09:38,683 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29600.11 MB 2025-02-14 10:09:38,683 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38013.63 MB 2025-02-14 10:09:38,683 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8413.52 MB 2025-02-14 10:09:38,683 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43241.18 MB 2025-02-14 10:09:38,683 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51604.62 MB 2025-02-14 10:09:38,683 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8363.44 MB 2025-02-14 10:09:38,683 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38013.63 MB 2025-02-14 10:09:38,845 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7929] 2025-02-14 10:09:38,846 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:09:38,846 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 10:09:38,847 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:09:38,847 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 10:09:38,852 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 10:09:38,853 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:09:38,853 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 10:09:38,853 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 10:11:43,950 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:11:43,951 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 10:11:43,956 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 10:11:43,960 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:11:43,960 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 195, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 10:11:43,961 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:11:43,961 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 195, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 10:11:46,975 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 10:11:46,975 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 10:11:46,975 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.01 seconds 2025-02-14 10:11:46,975 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:11:46,975 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22391.64 MB 2025-02-14 10:11:46,975 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23081.74 MB 2025-02-14 10:11:46,975 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 690.09 MB 2025-02-14 10:11:46,975 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59968.06 MB 2025-02-14 10:11:46,975 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26696.74 MB 2025-02-14 10:11:46,975 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33271.32 MB 2025-02-14 10:11:46,975 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32090.31 MB 2025-02-14 10:11:46,990 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 10:11:46,990 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 10:11:46,990 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:11:46,990 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:11:46,990 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23081.74 MB 2025-02-14 10:11:46,990 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23387.99 MB 2025-02-14 10:11:46,990 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 306.26 MB 2025-02-14 10:11:46,990 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26696.74 MB 2025-02-14 10:11:46,990 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27703.38 MB 2025-02-14 10:11:46,990 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1006.63 MB 2025-02-14 10:11:46,990 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25775.21 MB 2025-02-14 10:11:47,891 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 10:11:47,892 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 10:11:47,892 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.90 seconds 2025-02-14 10:11:47,892 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:11:47,892 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23387.99 MB 2025-02-14 10:11:47,892 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23641.47 MB 2025-02-14 10:11:47,892 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 253.48 MB 2025-02-14 10:11:47,892 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27703.38 MB 2025-02-14 10:11:47,892 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27703.38 MB 2025-02-14 10:11:47,892 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:11:47,892 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27581.08 MB 2025-02-14 10:11:47,900 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 10:11:47,900 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 10:11:47,900 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:11:47,900 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:11:47,900 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23641.41 MB 2025-02-14 10:11:47,900 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24543.44 MB 2025-02-14 10:11:47,900 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 902.03 MB 2025-02-14 10:11:47,900 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27703.38 MB 2025-02-14 10:11:47,900 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27703.38 MB 2025-02-14 10:11:47,900 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:11:47,900 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25220.27 MB 2025-02-14 10:11:48,003 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 10:11:48,003 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 10:11:48,003 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 10:11:48,003 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:11:48,003 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24543.44 MB 2025-02-14 10:11:48,003 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25614.45 MB 2025-02-14 10:11:48,003 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1071.01 MB 2025-02-14 10:11:48,003 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27703.38 MB 2025-02-14 10:11:48,003 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29733.42 MB 2025-02-14 10:11:48,003 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2030.04 MB 2025-02-14 10:11:48,003 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28262.86 MB 2025-02-14 10:11:48,004 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 10:11:48,004 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 10:11:48,004 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 10:11:48,004 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:11:48,004 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23641.41 MB 2025-02-14 10:11:48,004 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25614.45 MB 2025-02-14 10:11:48,004 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1973.04 MB 2025-02-14 10:11:48,004 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27703.38 MB 2025-02-14 10:11:48,004 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29733.42 MB 2025-02-14 10:11:48,004 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2030.04 MB 2025-02-14 10:11:48,004 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28262.86 MB 2025-02-14 10:11:48,120 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 10:11:48,120 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 10:11:48,120 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 10:11:48,120 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:11:48,120 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26346.72 MB 2025-02-14 10:11:48,120 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26713.22 MB 2025-02-14 10:11:48,120 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 366.51 MB 2025-02-14 10:11:48,120 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29733.42 MB 2025-02-14 10:11:48,120 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29926.36 MB 2025-02-14 10:11:48,120 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 192.94 MB 2025-02-14 10:11:48,120 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27055.03 MB 2025-02-14 10:11:48,130 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 10:11:48,130 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 10:11:48,130 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:11:48,130 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:11:48,131 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26910.38 MB 2025-02-14 10:11:48,131 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27133.15 MB 2025-02-14 10:11:48,131 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 222.76 MB 2025-02-14 10:11:48,131 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29926.36 MB 2025-02-14 10:11:48,131 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29926.36 MB 2025-02-14 10:11:48,131 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:11:48,131 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27177.26 MB 2025-02-14 10:11:48,132 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 10:11:48,132 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 10:11:48,132 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.17 seconds 2025-02-14 10:11:48,132 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:11:48,132 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21712.25 MB 2025-02-14 10:11:48,132 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27333.87 MB 2025-02-14 10:11:48,132 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5621.63 MB 2025-02-14 10:11:48,132 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59968.06 MB 2025-02-14 10:11:48,132 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29926.36 MB 2025-02-14 10:11:48,132 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30041.70 MB 2025-02-14 10:11:48,132 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27333.87 MB 2025-02-14 10:11:48,396 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 10:11:48,396 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 10:11:48,396 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 10:11:48,396 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:11:48,396 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27333.87 MB 2025-02-14 10:11:48,396 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25724.96 MB 2025-02-14 10:11:48,396 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1608.91 MB 2025-02-14 10:11:48,396 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29926.36 MB 2025-02-14 10:11:48,396 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29926.36 MB 2025-02-14 10:11:48,396 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:11:48,396 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27333.88 MB 2025-02-14 10:11:48,414 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8148, cut from 8150 2025-02-14 10:11:48,414 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 10:11:48,420 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 10:11:48,420 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 10:11:48,420 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:11:48,420 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:11:48,420 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25724.96 MB 2025-02-14 10:11:48,420 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34149.91 MB 2025-02-14 10:11:48,420 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8424.95 MB 2025-02-14 10:11:48,420 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29926.36 MB 2025-02-14 10:11:48,420 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40397.44 MB 2025-02-14 10:11:48,420 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10471.08 MB 2025-02-14 10:11:48,420 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34149.91 MB 2025-02-14 10:11:48,582 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7940] 2025-02-14 10:11:48,583 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:11:48,583 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 10:11:48,584 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:11:48,584 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 10:11:48,589 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 10:11:48,590 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:11:48,590 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 10:11:48,590 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 10:12:38,274 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:12:38,274 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 10:12:38,279 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 10:12:38,282 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:12:38,282 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2569, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 10:12:38,283 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:12:38,283 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2569, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 10:13:17,668 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 10:13:17,668 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 10:13:17,668 - resource_logging.py:150 - __exit__ - DEBUG - Time: 39.37 seconds 2025-02-14 10:13:17,668 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:13:17,668 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38935.76 MB 2025-02-14 10:13:17,668 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48027.31 MB 2025-02-14 10:13:17,668 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 9091.55 MB 2025-02-14 10:13:17,668 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 66678.95 MB 2025-02-14 10:13:17,668 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53693.38 MB 2025-02-14 10:13:17,668 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12985.57 MB 2025-02-14 10:13:17,668 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 57118.85 MB 2025-02-14 10:13:17,833 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 10:13:17,833 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 10:13:17,833 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 10:13:17,834 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:13:17,834 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 48027.31 MB 2025-02-14 10:13:17,834 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37199.33 MB 2025-02-14 10:13:17,834 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10827.98 MB 2025-02-14 10:13:17,834 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53693.38 MB 2025-02-14 10:13:17,834 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 84978.70 MB 2025-02-14 10:13:17,834 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 31285.31 MB 2025-02-14 10:13:17,834 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 73866.31 MB 2025-02-14 10:13:19,781 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 10:13:19,781 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 10:13:19,781 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.95 seconds 2025-02-14 10:13:19,781 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:13:19,781 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37199.33 MB 2025-02-14 10:13:19,781 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37730.17 MB 2025-02-14 10:13:19,781 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 10:13:19,781 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 84978.70 MB 2025-02-14 10:13:19,781 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42177.92 MB 2025-02-14 10:13:19,781 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -42800.78 MB 2025-02-14 10:13:19,781 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41710.54 MB 2025-02-14 10:13:19,795 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 10:13:19,795 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 10:13:19,795 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:13:19,795 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:13:19,795 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37730.17 MB 2025-02-14 10:13:19,795 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39619.52 MB 2025-02-14 10:13:19,795 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.35 MB 2025-02-14 10:13:19,795 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42177.92 MB 2025-02-14 10:13:19,795 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44065.36 MB 2025-02-14 10:13:19,795 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 10:13:19,795 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41036.95 MB 2025-02-14 10:13:20,003 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 10:13:20,003 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 10:13:20,003 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 10:13:20,003 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:13:20,003 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39619.52 MB 2025-02-14 10:13:20,003 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41861.38 MB 2025-02-14 10:13:20,003 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 10:13:20,003 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44065.36 MB 2025-02-14 10:13:20,003 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50199.53 MB 2025-02-14 10:13:20,003 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 10:13:20,003 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47406.56 MB 2025-02-14 10:13:20,004 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 10:13:20,004 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 10:13:20,004 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 10:13:20,004 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:13:20,004 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37730.17 MB 2025-02-14 10:13:20,004 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41861.38 MB 2025-02-14 10:13:20,004 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.21 MB 2025-02-14 10:13:20,004 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42177.92 MB 2025-02-14 10:13:20,004 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50199.53 MB 2025-02-14 10:13:20,004 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-14 10:13:20,004 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47406.56 MB 2025-02-14 10:13:20,174 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 10:13:20,174 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 10:13:20,174 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 10:13:20,174 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:13:20,174 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43395.82 MB 2025-02-14 10:13:20,174 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44162.82 MB 2025-02-14 10:13:20,174 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 10:13:20,174 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50199.53 MB 2025-02-14 10:13:20,174 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50612.67 MB 2025-02-14 10:13:20,174 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 10:13:20,174 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44870.61 MB 2025-02-14 10:13:20,193 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 10:13:20,193 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 10:13:20,193 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:13:20,194 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:13:20,194 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44575.71 MB 2025-02-14 10:13:20,194 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44804.33 MB 2025-02-14 10:13:20,194 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.62 MB 2025-02-14 10:13:20,194 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50612.67 MB 2025-02-14 10:13:20,194 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50612.67 MB 2025-02-14 10:13:20,194 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:13:20,194 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45024.06 MB 2025-02-14 10:13:20,195 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 10:13:20,195 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 10:13:20,195 - resource_logging.py:150 - __exit__ - DEBUG - Time: 41.91 seconds 2025-02-14 10:13:20,195 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:13:20,195 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29984.31 MB 2025-02-14 10:13:20,195 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45004.86 MB 2025-02-14 10:13:20,195 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 15020.55 MB 2025-02-14 10:13:20,195 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57726.21 MB 2025-02-14 10:13:20,195 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50612.67 MB 2025-02-14 10:13:20,195 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7113.54 MB 2025-02-14 10:13:20,195 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45024.06 MB 2025-02-14 10:13:20,465 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 10:13:20,465 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 10:13:20,465 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 10:13:20,465 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:13:20,465 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45004.86 MB 2025-02-14 10:13:20,465 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34981.48 MB 2025-02-14 10:13:20,465 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10023.37 MB 2025-02-14 10:13:20,465 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50612.67 MB 2025-02-14 10:13:20,465 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50612.67 MB 2025-02-14 10:13:20,465 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:13:20,465 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47509.77 MB 2025-02-14 10:13:20,483 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8140, cut from 8142 2025-02-14 10:13:20,484 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 10:13:20,489 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 10:13:20,489 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 10:13:20,489 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:13:20,489 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:13:20,489 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34981.48 MB 2025-02-14 10:13:20,489 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43398.09 MB 2025-02-14 10:13:20,489 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8416.60 MB 2025-02-14 10:13:20,489 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50612.67 MB 2025-02-14 10:13:20,490 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54796.48 MB 2025-02-14 10:13:20,490 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4183.82 MB 2025-02-14 10:13:20,490 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43398.09 MB 2025-02-14 10:13:20,651 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7932] 2025-02-14 10:13:20,652 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:13:20,652 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 10:13:20,653 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:13:20,653 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 10:13:20,658 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 10:13:20,659 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:13:20,659 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 10:13:20,659 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 10:13:28,692 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:13:28,692 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 10:13:28,697 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 10:13:28,700 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:13:28,700 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1162, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 10:13:28,701 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:13:28,701 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1162, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 10:13:46,855 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 10:13:46,855 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 10:13:46,855 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.15 seconds 2025-02-14 10:13:46,855 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:13:46,855 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29129.85 MB 2025-02-14 10:13:46,855 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33242.37 MB 2025-02-14 10:13:46,855 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4112.52 MB 2025-02-14 10:13:46,855 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63164.12 MB 2025-02-14 10:13:46,855 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39397.10 MB 2025-02-14 10:13:46,855 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23767.02 MB 2025-02-14 10:13:46,855 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42225.91 MB 2025-02-14 10:13:46,929 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 10:13:46,929 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 10:13:46,929 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 10:13:46,929 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:13:46,929 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33242.37 MB 2025-02-14 10:13:46,929 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29883.92 MB 2025-02-14 10:13:46,929 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3358.45 MB 2025-02-14 10:13:46,929 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39397.10 MB 2025-02-14 10:13:46,929 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52084.87 MB 2025-02-14 10:13:46,929 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 12687.77 MB 2025-02-14 10:13:46,929 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45664.61 MB 2025-02-14 10:13:48,861 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 10:13:48,861 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 10:13:48,861 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 10:13:48,861 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:13:48,861 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29883.92 MB 2025-02-14 10:13:48,861 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30414.76 MB 2025-02-14 10:13:48,861 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 10:13:48,861 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52084.87 MB 2025-02-14 10:13:48,861 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37409.00 MB 2025-02-14 10:13:48,861 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14675.87 MB 2025-02-14 10:13:48,861 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34394.10 MB 2025-02-14 10:13:48,874 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 10:13:48,874 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 10:13:48,874 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:13:48,874 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:13:48,874 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30414.76 MB 2025-02-14 10:13:48,874 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32304.12 MB 2025-02-14 10:13:48,874 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.35 MB 2025-02-14 10:13:48,874 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37409.00 MB 2025-02-14 10:13:48,874 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37409.00 MB 2025-02-14 10:13:48,874 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:13:48,874 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33721.54 MB 2025-02-14 10:13:49,089 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 10:13:49,089 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 10:13:49,089 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 10:13:49,090 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:13:49,090 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32304.12 MB 2025-02-14 10:13:49,090 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34545.97 MB 2025-02-14 10:13:49,090 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 10:13:49,090 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37409.00 MB 2025-02-14 10:13:49,090 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43071.31 MB 2025-02-14 10:13:49,090 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 10:13:49,090 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40091.15 MB 2025-02-14 10:13:49,090 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 10:13:49,090 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 10:13:49,090 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 10:13:49,090 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:13:49,090 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30414.76 MB 2025-02-14 10:13:49,090 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34545.97 MB 2025-02-14 10:13:49,090 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.21 MB 2025-02-14 10:13:49,090 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37409.00 MB 2025-02-14 10:13:49,090 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43071.31 MB 2025-02-14 10:13:49,090 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 10:13:49,090 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40091.15 MB 2025-02-14 10:13:49,262 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 10:13:49,262 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 10:13:49,262 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 10:13:49,262 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:13:49,263 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36080.41 MB 2025-02-14 10:13:49,263 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36847.42 MB 2025-02-14 10:13:49,263 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 10:13:49,263 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43071.31 MB 2025-02-14 10:13:49,263 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43484.45 MB 2025-02-14 10:13:49,263 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 10:13:49,263 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37555.20 MB 2025-02-14 10:13:49,281 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 10:13:49,281 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 10:13:49,281 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:13:49,282 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:13:49,282 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37260.30 MB 2025-02-14 10:13:49,282 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37488.41 MB 2025-02-14 10:13:49,282 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.10 MB 2025-02-14 10:13:49,282 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43484.45 MB 2025-02-14 10:13:49,282 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43484.45 MB 2025-02-14 10:13:49,282 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:13:49,282 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37710.95 MB 2025-02-14 10:13:49,283 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 10:13:49,283 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 10:13:49,283 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.58 seconds 2025-02-14 10:13:49,283 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:13:49,283 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25081.35 MB 2025-02-14 10:13:49,283 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37688.35 MB 2025-02-14 10:13:49,283 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12606.99 MB 2025-02-14 10:13:49,283 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63164.12 MB 2025-02-14 10:13:49,283 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43484.45 MB 2025-02-14 10:13:49,283 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19679.67 MB 2025-02-14 10:13:49,283 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37710.95 MB 2025-02-14 10:13:49,553 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 10:13:49,553 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 10:13:49,553 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 10:13:49,553 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:13:49,553 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37688.35 MB 2025-02-14 10:13:49,553 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30069.68 MB 2025-02-14 10:13:49,553 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7618.66 MB 2025-02-14 10:13:49,553 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43484.45 MB 2025-02-14 10:13:49,553 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43484.45 MB 2025-02-14 10:13:49,553 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:13:49,553 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40185.88 MB 2025-02-14 10:13:49,571 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8116, cut from 8118 2025-02-14 10:13:49,571 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 10:13:49,577 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 10:13:49,577 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 10:13:49,577 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:13:49,577 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:13:49,577 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30069.68 MB 2025-02-14 10:13:49,577 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38462.11 MB 2025-02-14 10:13:49,577 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8392.42 MB 2025-02-14 10:13:49,577 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43484.45 MB 2025-02-14 10:13:49,577 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51826.92 MB 2025-02-14 10:13:49,577 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8342.47 MB 2025-02-14 10:13:49,577 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38462.11 MB 2025-02-14 10:13:49,738 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7908] 2025-02-14 10:13:49,739 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:13:49,739 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 10:13:49,740 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:13:49,740 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 10:13:49,745 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 10:13:49,746 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:13:49,746 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 10:13:49,746 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 10:14:09,311 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:14:09,311 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 10:14:09,316 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 10:14:09,319 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:14:09,319 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 145, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 10:14:09,320 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:14:09,320 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 145, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 10:14:11,612 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 10:14:11,612 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 10:14:11,612 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.29 seconds 2025-02-14 10:14:11,612 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:14:11,612 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22043.24 MB 2025-02-14 10:14:11,612 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22556.38 MB 2025-02-14 10:14:11,612 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 513.15 MB 2025-02-14 10:14:11,612 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60169.39 MB 2025-02-14 10:14:11,612 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26692.55 MB 2025-02-14 10:14:11,612 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33476.84 MB 2025-02-14 10:14:11,612 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31515.41 MB 2025-02-14 10:14:11,624 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 10:14:11,624 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 10:14:11,624 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:14:11,625 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:14:11,625 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22556.38 MB 2025-02-14 10:14:11,625 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22805.00 MB 2025-02-14 10:14:11,625 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 248.62 MB 2025-02-14 10:14:11,625 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26692.55 MB 2025-02-14 10:14:11,625 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26692.55 MB 2025-02-14 10:14:11,625 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:14:11,625 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24596.66 MB 2025-02-14 10:14:12,337 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 10:14:12,338 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 10:14:12,338 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.71 seconds 2025-02-14 10:14:12,338 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:14:12,338 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22805.00 MB 2025-02-14 10:14:12,338 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22997.43 MB 2025-02-14 10:14:12,338 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 192.43 MB 2025-02-14 10:14:12,338 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26692.55 MB 2025-02-14 10:14:12,338 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26692.55 MB 2025-02-14 10:14:12,338 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:14:12,338 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26976.48 MB 2025-02-14 10:14:12,345 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 10:14:12,345 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 10:14:12,345 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 10:14:12,345 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:14:12,345 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22997.37 MB 2025-02-14 10:14:12,345 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23682.16 MB 2025-02-14 10:14:12,345 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 684.79 MB 2025-02-14 10:14:12,345 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26692.55 MB 2025-02-14 10:14:12,345 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26692.55 MB 2025-02-14 10:14:12,345 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:14:12,345 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24195.98 MB 2025-02-14 10:14:12,427 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 10:14:12,427 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 10:14:12,427 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 10:14:12,427 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:14:12,427 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23682.16 MB 2025-02-14 10:14:12,427 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24494.87 MB 2025-02-14 10:14:12,427 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 812.71 MB 2025-02-14 10:14:12,427 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26692.55 MB 2025-02-14 10:14:12,427 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27896.32 MB 2025-02-14 10:14:12,427 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1203.77 MB 2025-02-14 10:14:12,427 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26505.55 MB 2025-02-14 10:14:12,428 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 10:14:12,428 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 10:14:12,428 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 10:14:12,428 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:14:12,428 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22997.37 MB 2025-02-14 10:14:12,428 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24494.87 MB 2025-02-14 10:14:12,428 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1497.50 MB 2025-02-14 10:14:12,428 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26692.55 MB 2025-02-14 10:14:12,428 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27896.32 MB 2025-02-14 10:14:12,428 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1203.77 MB 2025-02-14 10:14:12,428 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26505.55 MB 2025-02-14 10:14:12,490 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 10:14:12,490 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 10:14:12,490 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 10:14:12,491 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:14:12,491 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25050.78 MB 2025-02-14 10:14:12,491 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25329.73 MB 2025-02-14 10:14:12,491 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 278.96 MB 2025-02-14 10:14:12,491 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27896.32 MB 2025-02-14 10:14:12,491 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28045.21 MB 2025-02-14 10:14:12,491 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 148.90 MB 2025-02-14 10:14:12,491 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25597.35 MB 2025-02-14 10:14:12,499 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 10:14:12,499 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 10:14:12,499 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:14:12,499 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:14:12,499 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25479.41 MB 2025-02-14 10:14:12,499 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25708.33 MB 2025-02-14 10:14:12,499 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.92 MB 2025-02-14 10:14:12,499 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28045.21 MB 2025-02-14 10:14:12,499 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28045.21 MB 2025-02-14 10:14:12,499 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:14:12,499 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25711.99 MB 2025-02-14 10:14:12,500 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 10:14:12,500 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 10:14:12,500 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.18 seconds 2025-02-14 10:14:12,500 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:14:12,500 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21538.04 MB 2025-02-14 10:14:12,500 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25909.19 MB 2025-02-14 10:14:12,501 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4371.14 MB 2025-02-14 10:14:12,501 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60169.39 MB 2025-02-14 10:14:12,501 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28045.21 MB 2025-02-14 10:14:12,501 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32124.17 MB 2025-02-14 10:14:12,501 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25909.19 MB 2025-02-14 10:14:12,768 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 10:14:12,768 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 10:14:12,768 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 10:14:12,768 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:14:12,768 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25909.19 MB 2025-02-14 10:14:12,768 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25336.43 MB 2025-02-14 10:14:12,768 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -572.75 MB 2025-02-14 10:14:12,768 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28045.21 MB 2025-02-14 10:14:12,768 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28045.21 MB 2025-02-14 10:14:12,768 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:14:12,768 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27013.10 MB 2025-02-14 10:14:12,786 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8153, cut from 8155 2025-02-14 10:14:12,787 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2,'] 2025-02-14 10:14:12,793 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 10:14:12,793 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 10:14:12,793 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:14:12,793 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:14:12,793 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25336.43 MB 2025-02-14 10:14:12,793 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33766.83 MB 2025-02-14 10:14:12,793 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8430.40 MB 2025-02-14 10:14:12,793 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28045.21 MB 2025-02-14 10:14:12,793 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38520.49 MB 2025-02-14 10:14:12,793 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10475.27 MB 2025-02-14 10:14:12,793 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33766.83 MB 2025-02-14 10:14:12,955 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7945] 2025-02-14 10:14:12,956 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:14:12,956 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 10:14:12,957 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:14:12,957 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 10:14:12,962 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 10:14:12,963 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:14:12,963 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 10:14:12,963 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2,'] 2025-02-14 10:16:01,137 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:16:01,138 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 10:16:01,142 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 10:16:01,147 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:16:01,147 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 427, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 10:16:01,148 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:16:01,148 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 427, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 10:16:07,686 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 10:16:07,687 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 10:16:07,687 - resource_logging.py:150 - __exit__ - DEBUG - Time: 6.53 seconds 2025-02-14 10:16:07,687 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:16:07,687 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24008.26 MB 2025-02-14 10:16:07,687 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25520.30 MB 2025-02-14 10:16:07,687 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1512.05 MB 2025-02-14 10:16:07,687 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46900.71 MB 2025-02-14 10:16:07,687 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30299.65 MB 2025-02-14 10:16:07,687 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16601.06 MB 2025-02-14 10:16:07,687 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34386.40 MB 2025-02-14 10:16:07,719 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 10:16:07,719 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 10:16:07,719 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 10:16:07,719 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:16:07,719 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25520.30 MB 2025-02-14 10:16:07,719 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26062.88 MB 2025-02-14 10:16:07,719 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 542.58 MB 2025-02-14 10:16:07,719 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30299.65 MB 2025-02-14 10:16:07,719 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36773.56 MB 2025-02-14 10:16:07,719 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6473.91 MB 2025-02-14 10:16:07,719 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32784.82 MB 2025-02-14 10:16:09,627 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 10:16:09,627 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 10:16:09,627 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 10:16:09,627 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:16:09,627 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26062.88 MB 2025-02-14 10:16:09,627 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26593.73 MB 2025-02-14 10:16:09,627 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 10:16:09,627 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36773.56 MB 2025-02-14 10:16:09,627 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30912.02 MB 2025-02-14 10:16:09,627 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5861.54 MB 2025-02-14 10:16:09,627 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30573.06 MB 2025-02-14 10:16:09,641 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 10:16:09,641 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 10:16:09,641 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:16:09,641 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:16:09,641 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26593.73 MB 2025-02-14 10:16:09,641 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28483.08 MB 2025-02-14 10:16:09,641 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.35 MB 2025-02-14 10:16:09,641 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30912.02 MB 2025-02-14 10:16:09,641 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32799.46 MB 2025-02-14 10:16:09,641 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 10:16:09,641 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29900.51 MB 2025-02-14 10:16:09,844 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 10:16:09,845 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 10:16:09,845 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 10:16:09,845 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:16:09,845 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28483.08 MB 2025-02-14 10:16:09,845 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30724.94 MB 2025-02-14 10:16:09,845 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 10:16:09,845 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32799.46 MB 2025-02-14 10:16:09,845 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38933.63 MB 2025-02-14 10:16:09,845 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 10:16:09,845 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36270.12 MB 2025-02-14 10:16:09,845 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 10:16:09,845 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 10:16:09,845 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 10:16:09,845 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:16:09,845 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26593.73 MB 2025-02-14 10:16:09,845 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30724.94 MB 2025-02-14 10:16:09,845 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.21 MB 2025-02-14 10:16:09,845 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30912.02 MB 2025-02-14 10:16:09,845 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38933.63 MB 2025-02-14 10:16:09,845 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-14 10:16:09,846 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36270.12 MB 2025-02-14 10:16:10,005 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 10:16:10,005 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 10:16:10,005 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 10:16:10,005 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:16:10,006 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32259.38 MB 2025-02-14 10:16:10,006 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33026.38 MB 2025-02-14 10:16:10,006 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 10:16:10,006 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38933.63 MB 2025-02-14 10:16:10,006 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39348.86 MB 2025-02-14 10:16:10,006 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 10:16:10,006 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33734.17 MB 2025-02-14 10:16:10,024 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 10:16:10,024 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 10:16:10,024 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:16:10,024 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:16:10,024 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33439.27 MB 2025-02-14 10:16:10,024 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33668.06 MB 2025-02-14 10:16:10,024 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.79 MB 2025-02-14 10:16:10,024 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39348.86 MB 2025-02-14 10:16:10,024 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39348.86 MB 2025-02-14 10:16:10,024 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:16:10,024 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33871.32 MB 2025-02-14 10:16:10,025 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 10:16:10,025 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 10:16:10,025 - resource_logging.py:150 - __exit__ - DEBUG - Time: 8.88 seconds 2025-02-14 10:16:10,025 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:16:10,025 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22520.55 MB 2025-02-14 10:16:10,025 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33868.91 MB 2025-02-14 10:16:10,025 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11348.36 MB 2025-02-14 10:16:10,025 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46900.71 MB 2025-02-14 10:16:10,025 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39348.86 MB 2025-02-14 10:16:10,025 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7551.84 MB 2025-02-14 10:16:10,025 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33871.32 MB 2025-02-14 10:16:10,291 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 10:16:10,291 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 10:16:10,291 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 10:16:10,291 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:16:10,291 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33868.91 MB 2025-02-14 10:16:10,291 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27523.19 MB 2025-02-14 10:16:10,291 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6345.72 MB 2025-02-14 10:16:10,291 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39348.86 MB 2025-02-14 10:16:10,291 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39348.86 MB 2025-02-14 10:16:10,291 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:16:10,291 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36377.81 MB 2025-02-14 10:16:10,318 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8153, cut from 8155 2025-02-14 10:16:10,319 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 10:16:10,338 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 10:16:10,338 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 10:16:10,339 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-14 10:16:10,339 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:16:10,339 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27523.19 MB 2025-02-14 10:16:10,339 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35953.59 MB 2025-02-14 10:16:10,339 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8430.40 MB 2025-02-14 10:16:10,339 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39348.86 MB 2025-02-14 10:16:10,339 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49824.14 MB 2025-02-14 10:16:10,339 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10475.27 MB 2025-02-14 10:16:10,339 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35953.59 MB 2025-02-14 10:16:10,524 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7945] 2025-02-14 10:16:10,526 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:16:10,527 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 10:16:10,528 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:16:10,529 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 10:16:10,535 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 10:16:10,537 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:16:10,537 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 10:16:10,537 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 10:17:27,840 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:17:27,841 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 10:17:27,846 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 10:17:27,851 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:17:27,851 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2451, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 10:17:27,852 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:17:27,852 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2451, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 10:18:05,807 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 10:18:05,807 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 10:18:05,807 - resource_logging.py:150 - __exit__ - DEBUG - Time: 37.94 seconds 2025-02-14 10:18:05,807 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:18:05,807 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38112.19 MB 2025-02-14 10:18:05,807 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46786.14 MB 2025-02-14 10:18:05,807 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8673.95 MB 2025-02-14 10:18:05,807 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 75287.76 MB 2025-02-14 10:18:05,807 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52451.87 MB 2025-02-14 10:18:05,807 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22835.89 MB 2025-02-14 10:18:05,808 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 55738.09 MB 2025-02-14 10:18:05,967 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 10:18:05,967 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 10:18:05,967 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 10:18:05,967 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:18:05,967 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46786.14 MB 2025-02-14 10:18:05,967 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36585.41 MB 2025-02-14 10:18:05,967 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10200.73 MB 2025-02-14 10:18:05,967 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52451.87 MB 2025-02-14 10:18:05,967 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 81392.57 MB 2025-02-14 10:18:05,967 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 28940.70 MB 2025-02-14 10:18:05,967 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 70839.48 MB 2025-02-14 10:18:07,919 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 10:18:07,919 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 10:18:07,920 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.95 seconds 2025-02-14 10:18:07,920 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:18:07,920 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36585.41 MB 2025-02-14 10:18:07,920 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37116.25 MB 2025-02-14 10:18:07,920 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 10:18:07,920 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 81392.57 MB 2025-02-14 10:18:07,920 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41561.36 MB 2025-02-14 10:18:07,920 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -39831.21 MB 2025-02-14 10:18:07,920 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41095.58 MB 2025-02-14 10:18:07,933 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 10:18:07,933 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 10:18:07,933 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:18:07,933 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:18:07,933 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37116.25 MB 2025-02-14 10:18:07,933 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39005.60 MB 2025-02-14 10:18:07,933 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.35 MB 2025-02-14 10:18:07,933 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41561.36 MB 2025-02-14 10:18:07,933 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43448.80 MB 2025-02-14 10:18:07,933 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 10:18:07,933 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40423.03 MB 2025-02-14 10:18:08,146 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 10:18:08,146 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 10:18:08,146 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 10:18:08,146 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:18:08,146 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39005.60 MB 2025-02-14 10:18:08,146 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41247.46 MB 2025-02-14 10:18:08,146 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 10:18:08,146 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43448.80 MB 2025-02-14 10:18:08,146 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49582.96 MB 2025-02-14 10:18:08,146 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 10:18:08,146 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46792.64 MB 2025-02-14 10:18:08,147 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 10:18:08,147 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 10:18:08,147 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 10:18:08,147 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:18:08,147 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37116.25 MB 2025-02-14 10:18:08,147 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41247.46 MB 2025-02-14 10:18:08,147 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.21 MB 2025-02-14 10:18:08,147 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41561.36 MB 2025-02-14 10:18:08,147 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49582.96 MB 2025-02-14 10:18:08,147 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-14 10:18:08,147 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46792.64 MB 2025-02-14 10:18:08,317 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 10:18:08,317 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 10:18:08,317 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 10:18:08,317 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:18:08,317 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42781.90 MB 2025-02-14 10:18:08,317 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43548.90 MB 2025-02-14 10:18:08,317 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 10:18:08,317 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49582.96 MB 2025-02-14 10:18:08,317 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49998.20 MB 2025-02-14 10:18:08,317 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 10:18:08,317 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44256.69 MB 2025-02-14 10:18:08,337 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 10:18:08,337 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 10:18:08,337 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:18:08,337 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:18:08,337 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43961.79 MB 2025-02-14 10:18:08,337 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44189.75 MB 2025-02-14 10:18:08,337 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.96 MB 2025-02-14 10:18:08,337 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49998.20 MB 2025-02-14 10:18:08,337 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49998.20 MB 2025-02-14 10:18:08,337 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:18:08,337 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44401.67 MB 2025-02-14 10:18:08,338 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 10:18:08,338 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 10:18:08,338 - resource_logging.py:150 - __exit__ - DEBUG - Time: 40.48 seconds 2025-02-14 10:18:08,338 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:18:08,338 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29572.52 MB 2025-02-14 10:18:08,338 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44389.69 MB 2025-02-14 10:18:08,338 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14817.17 MB 2025-02-14 10:18:08,338 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 66746.06 MB 2025-02-14 10:18:08,338 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49998.20 MB 2025-02-14 10:18:08,338 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16747.86 MB 2025-02-14 10:18:08,338 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44401.67 MB 2025-02-14 10:18:08,608 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 10:18:08,608 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 10:18:08,608 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 10:18:08,608 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:18:08,608 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44389.69 MB 2025-02-14 10:18:08,608 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34560.79 MB 2025-02-14 10:18:08,608 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9828.91 MB 2025-02-14 10:18:08,608 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49998.20 MB 2025-02-14 10:18:08,608 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49998.20 MB 2025-02-14 10:18:08,608 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:18:08,608 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46887.23 MB 2025-02-14 10:18:08,626 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8116, cut from 8118 2025-02-14 10:18:08,626 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 10:18:08,632 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 10:18:08,632 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 10:18:08,632 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:18:08,632 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:18:08,632 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34560.79 MB 2025-02-14 10:18:08,632 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42952.35 MB 2025-02-14 10:18:08,632 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8391.56 MB 2025-02-14 10:18:08,632 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49998.20 MB 2025-02-14 10:18:08,632 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54169.44 MB 2025-02-14 10:18:08,632 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4171.24 MB 2025-02-14 10:18:08,632 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42952.35 MB 2025-02-14 10:18:08,795 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7908] 2025-02-14 10:18:08,797 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:18:08,797 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 10:18:08,798 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:18:08,798 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 10:18:08,803 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 10:18:08,804 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:18:08,804 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 10:18:08,804 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 10:18:18,693 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:18:18,693 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 10:18:18,698 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 10:18:18,702 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:18:18,702 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1973, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 10:18:18,703 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:18:18,703 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1973, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 10:18:49,507 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 10:18:49,507 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 10:18:49,507 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.79 seconds 2025-02-14 10:18:49,507 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:18:49,507 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34781.03 MB 2025-02-14 10:18:49,507 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41763.37 MB 2025-02-14 10:18:49,507 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6982.34 MB 2025-02-14 10:18:49,507 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62511.91 MB 2025-02-14 10:18:49,507 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50392.47 MB 2025-02-14 10:18:49,507 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12119.44 MB 2025-02-14 10:18:49,507 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50595.18 MB 2025-02-14 10:18:49,661 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 10:18:49,661 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 10:18:49,661 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 10:18:49,661 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:18:49,661 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41763.37 MB 2025-02-14 10:18:49,661 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34099.01 MB 2025-02-14 10:18:49,661 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7664.36 MB 2025-02-14 10:18:49,661 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50392.47 MB 2025-02-14 10:18:49,661 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 70925.68 MB 2025-02-14 10:18:49,661 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 20533.22 MB 2025-02-14 10:18:49,661 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 62336.12 MB 2025-02-14 10:18:51,596 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 10:18:51,596 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 10:18:51,596 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 10:18:51,596 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:18:51,596 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34099.01 MB 2025-02-14 10:18:51,596 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34629.85 MB 2025-02-14 10:18:51,596 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 10:18:51,596 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 70925.68 MB 2025-02-14 10:18:51,596 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40653.29 MB 2025-02-14 10:18:51,596 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30272.39 MB 2025-02-14 10:18:51,596 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38609.18 MB 2025-02-14 10:18:51,612 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 10:18:51,612 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 10:18:51,612 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:18:51,612 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:18:51,612 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34629.85 MB 2025-02-14 10:18:51,612 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36519.20 MB 2025-02-14 10:18:51,612 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.35 MB 2025-02-14 10:18:51,612 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40653.29 MB 2025-02-14 10:18:51,612 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42540.73 MB 2025-02-14 10:18:51,612 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 10:18:51,612 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37936.63 MB 2025-02-14 10:18:51,824 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 10:18:51,824 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 10:18:51,824 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 10:18:51,824 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:18:51,824 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36519.20 MB 2025-02-14 10:18:51,824 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38761.06 MB 2025-02-14 10:18:51,824 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 10:18:51,824 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42540.73 MB 2025-02-14 10:18:51,824 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48203.04 MB 2025-02-14 10:18:51,824 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 10:18:51,824 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44306.24 MB 2025-02-14 10:18:51,824 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 10:18:51,824 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 10:18:51,824 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 10:18:51,824 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:18:51,824 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34629.85 MB 2025-02-14 10:18:51,825 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38761.06 MB 2025-02-14 10:18:51,825 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.21 MB 2025-02-14 10:18:51,825 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40653.29 MB 2025-02-14 10:18:51,825 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48203.04 MB 2025-02-14 10:18:51,825 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-14 10:18:51,825 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44306.24 MB 2025-02-14 10:18:51,992 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 10:18:51,992 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 10:18:51,992 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 10:18:51,992 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:18:51,992 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40295.50 MB 2025-02-14 10:18:51,992 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41062.50 MB 2025-02-14 10:18:51,992 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 10:18:51,992 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48203.04 MB 2025-02-14 10:18:51,992 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48618.27 MB 2025-02-14 10:18:51,992 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 10:18:51,992 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41770.29 MB 2025-02-14 10:18:52,011 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 10:18:52,011 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 10:18:52,011 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:18:52,011 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:18:52,011 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41475.39 MB 2025-02-14 10:18:52,011 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41703.57 MB 2025-02-14 10:18:52,011 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.17 MB 2025-02-14 10:18:52,011 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48618.27 MB 2025-02-14 10:18:52,011 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48618.27 MB 2025-02-14 10:18:52,011 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:18:52,011 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41927.47 MB 2025-02-14 10:18:52,012 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 10:18:52,012 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 10:18:52,012 - resource_logging.py:150 - __exit__ - DEBUG - Time: 33.31 seconds 2025-02-14 10:18:52,012 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:18:52,012 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27906.94 MB 2025-02-14 10:18:52,012 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41903.66 MB 2025-02-14 10:18:52,012 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13996.71 MB 2025-02-14 10:18:52,012 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62511.91 MB 2025-02-14 10:18:52,012 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48618.27 MB 2025-02-14 10:18:52,012 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13893.63 MB 2025-02-14 10:18:52,012 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41927.47 MB 2025-02-14 10:18:52,283 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 10:18:52,283 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 10:18:52,283 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 10:18:52,283 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:18:52,283 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41903.66 MB 2025-02-14 10:18:52,283 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32897.49 MB 2025-02-14 10:18:52,283 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9006.17 MB 2025-02-14 10:18:52,283 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48618.27 MB 2025-02-14 10:18:52,283 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48618.27 MB 2025-02-14 10:18:52,283 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:18:52,283 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44403.04 MB 2025-02-14 10:18:52,301 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8122, cut from 8124 2025-02-14 10:18:52,301 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 10:18:52,307 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 10:18:52,307 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 10:18:52,307 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:18:52,307 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:18:52,307 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32897.49 MB 2025-02-14 10:18:52,307 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41294.89 MB 2025-02-14 10:18:52,307 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8397.40 MB 2025-02-14 10:18:52,307 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48618.27 MB 2025-02-14 10:18:52,307 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56969.13 MB 2025-02-14 10:18:52,307 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8350.86 MB 2025-02-14 10:18:52,307 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41294.89 MB 2025-02-14 10:18:52,475 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7914] 2025-02-14 10:18:52,476 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:18:52,476 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 10:18:52,477 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:18:52,477 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 10:18:52,482 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 10:18:52,483 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:18:52,483 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 10:18:52,484 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 10:20:00,694 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:20:00,694 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 10:20:00,699 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 10:20:00,704 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:20:00,704 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 185, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 10:20:00,705 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:20:00,705 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 185, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 10:20:03,553 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 10:20:03,553 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 10:20:03,553 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.84 seconds 2025-02-14 10:20:03,553 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:20:03,553 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22321.96 MB 2025-02-14 10:20:03,553 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22976.67 MB 2025-02-14 10:20:03,553 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 654.70 MB 2025-02-14 10:20:03,553 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65319.99 MB 2025-02-14 10:20:03,553 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26692.55 MB 2025-02-14 10:20:03,553 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -38627.44 MB 2025-02-14 10:20:03,554 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31794.14 MB 2025-02-14 10:20:03,567 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 10:20:03,567 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 10:20:03,567 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:20:03,567 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:20:03,567 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22976.67 MB 2025-02-14 10:20:03,567 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23265.78 MB 2025-02-14 10:20:03,567 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 289.11 MB 2025-02-14 10:20:03,567 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26692.55 MB 2025-02-14 10:20:03,567 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27334.28 MB 2025-02-14 10:20:03,567 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 641.73 MB 2025-02-14 10:20:03,567 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25550.92 MB 2025-02-14 10:20:04,434 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 10:20:04,434 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 10:20:04,434 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.86 seconds 2025-02-14 10:20:04,434 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:20:04,434 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23265.78 MB 2025-02-14 10:20:04,434 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23505.98 MB 2025-02-14 10:20:04,434 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 240.21 MB 2025-02-14 10:20:04,434 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27334.28 MB 2025-02-14 10:20:04,434 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27334.28 MB 2025-02-14 10:20:04,434 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:20:04,434 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27437.25 MB 2025-02-14 10:20:04,442 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 10:20:04,442 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 10:20:04,442 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:20:04,442 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:20:04,442 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23505.92 MB 2025-02-14 10:20:04,442 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24360.72 MB 2025-02-14 10:20:04,442 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 854.81 MB 2025-02-14 10:20:04,442 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27334.28 MB 2025-02-14 10:20:04,442 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27334.28 MB 2025-02-14 10:20:04,442 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:20:04,442 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25002.12 MB 2025-02-14 10:20:04,540 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 10:20:04,540 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 10:20:04,540 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 10:20:04,540 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:20:04,540 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24360.72 MB 2025-02-14 10:20:04,540 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25375.20 MB 2025-02-14 10:20:04,540 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1014.48 MB 2025-02-14 10:20:04,540 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27334.28 MB 2025-02-14 10:20:04,540 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29901.19 MB 2025-02-14 10:20:04,540 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2566.91 MB 2025-02-14 10:20:04,540 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27887.88 MB 2025-02-14 10:20:04,541 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 10:20:04,541 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 10:20:04,541 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 10:20:04,541 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:20:04,541 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23505.92 MB 2025-02-14 10:20:04,541 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25375.20 MB 2025-02-14 10:20:04,541 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1869.28 MB 2025-02-14 10:20:04,541 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27334.28 MB 2025-02-14 10:20:04,541 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29901.19 MB 2025-02-14 10:20:04,541 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2566.91 MB 2025-02-14 10:20:04,541 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27887.88 MB 2025-02-14 10:20:04,618 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 10:20:04,618 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 10:20:04,618 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 10:20:04,619 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:20:04,619 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26069.13 MB 2025-02-14 10:20:04,619 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26416.98 MB 2025-02-14 10:20:04,619 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 347.85 MB 2025-02-14 10:20:04,619 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29901.19 MB 2025-02-14 10:20:04,619 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30087.84 MB 2025-02-14 10:20:04,619 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 186.65 MB 2025-02-14 10:20:04,619 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26742.39 MB 2025-02-14 10:20:04,629 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 10:20:04,629 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 10:20:04,629 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:20:04,629 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:20:04,629 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26603.82 MB 2025-02-14 10:20:04,629 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26819.99 MB 2025-02-14 10:20:04,629 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 216.16 MB 2025-02-14 10:20:04,629 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30087.84 MB 2025-02-14 10:20:04,629 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30087.84 MB 2025-02-14 10:20:04,629 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:20:04,629 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26859.92 MB 2025-02-14 10:20:04,630 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 10:20:04,630 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 10:20:04,630 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.92 seconds 2025-02-14 10:20:04,630 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:20:04,630 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21677.41 MB 2025-02-14 10:20:04,630 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27020.76 MB 2025-02-14 10:20:04,630 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5343.36 MB 2025-02-14 10:20:04,630 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65319.99 MB 2025-02-14 10:20:04,630 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30087.84 MB 2025-02-14 10:20:04,630 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35232.15 MB 2025-02-14 10:20:04,630 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27020.76 MB 2025-02-14 10:20:04,896 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 10:20:04,896 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 10:20:04,896 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 10:20:04,896 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:20:04,896 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27020.76 MB 2025-02-14 10:20:04,896 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25645.04 MB 2025-02-14 10:20:04,896 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1375.72 MB 2025-02-14 10:20:04,896 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30087.84 MB 2025-02-14 10:20:04,897 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30087.84 MB 2025-02-14 10:20:04,897 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:20:04,897 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27020.77 MB 2025-02-14 10:20:04,914 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8150, cut from 8152 2025-02-14 10:20:04,914 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 10:20:04,921 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 10:20:04,921 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 10:20:04,921 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:20:04,921 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:20:04,921 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25645.04 MB 2025-02-14 10:20:04,921 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34071.54 MB 2025-02-14 10:20:04,921 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8426.50 MB 2025-02-14 10:20:04,921 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30087.84 MB 2025-02-14 10:20:04,921 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40561.02 MB 2025-02-14 10:20:04,921 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10473.18 MB 2025-02-14 10:20:04,921 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34071.54 MB 2025-02-14 10:20:05,083 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7942] 2025-02-14 10:20:05,084 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:20:05,084 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 10:20:05,085 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:20:05,085 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 10:20:05,090 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 10:20:05,091 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:20:05,091 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 10:20:05,091 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 10:20:51,682 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:20:51,683 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 10:20:51,688 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 10:20:51,691 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:20:51,691 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1498, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 10:20:51,692 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:20:51,692 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1498, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 10:21:14,654 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 10:21:14,654 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 10:21:14,654 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.95 seconds 2025-02-14 10:21:14,654 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:21:14,654 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31471.16 MB 2025-02-14 10:21:14,654 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36772.76 MB 2025-02-14 10:21:14,654 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5301.60 MB 2025-02-14 10:21:14,654 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53127.15 MB 2025-02-14 10:21:14,654 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48748.30 MB 2025-02-14 10:21:14,654 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4378.85 MB 2025-02-14 10:21:14,654 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45699.67 MB 2025-02-14 10:21:14,737 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 10:21:14,737 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 10:21:14,737 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 10:21:14,737 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:21:14,737 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36772.76 MB 2025-02-14 10:21:14,737 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31629.63 MB 2025-02-14 10:21:14,737 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5143.12 MB 2025-02-14 10:21:14,737 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48748.30 MB 2025-02-14 10:21:14,737 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58275.66 MB 2025-02-14 10:21:14,737 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9527.36 MB 2025-02-14 10:21:14,737 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51132.62 MB 2025-02-14 10:21:16,666 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 10:21:16,666 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 10:21:16,666 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 10:21:16,666 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:21:16,666 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31629.63 MB 2025-02-14 10:21:16,666 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32160.47 MB 2025-02-14 10:21:16,666 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 10:21:16,667 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58275.66 MB 2025-02-14 10:21:16,667 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43446.70 MB 2025-02-14 10:21:16,667 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14828.96 MB 2025-02-14 10:21:16,667 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36139.81 MB 2025-02-14 10:21:16,680 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 10:21:16,680 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 10:21:16,680 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:21:16,680 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:21:16,680 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32160.47 MB 2025-02-14 10:21:16,680 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34049.83 MB 2025-02-14 10:21:16,680 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.35 MB 2025-02-14 10:21:16,680 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43446.70 MB 2025-02-14 10:21:16,680 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43446.70 MB 2025-02-14 10:21:16,680 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:21:16,680 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35467.25 MB 2025-02-14 10:21:16,891 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 10:21:16,891 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 10:21:16,891 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 10:21:16,891 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:21:16,891 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34049.83 MB 2025-02-14 10:21:16,891 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36291.68 MB 2025-02-14 10:21:16,891 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 10:21:16,891 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43446.70 MB 2025-02-14 10:21:16,891 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47221.57 MB 2025-02-14 10:21:16,891 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 10:21:16,891 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41836.86 MB 2025-02-14 10:21:16,892 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 10:21:16,892 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 10:21:16,892 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 10:21:16,892 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:21:16,892 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32160.47 MB 2025-02-14 10:21:16,892 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36291.68 MB 2025-02-14 10:21:16,892 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.21 MB 2025-02-14 10:21:16,892 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43446.70 MB 2025-02-14 10:21:16,892 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47221.57 MB 2025-02-14 10:21:16,892 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 10:21:16,892 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41836.86 MB 2025-02-14 10:21:17,062 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 10:21:17,062 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 10:21:17,062 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 10:21:17,062 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:21:17,062 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37826.12 MB 2025-02-14 10:21:17,062 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38593.13 MB 2025-02-14 10:21:17,062 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 10:21:17,062 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47221.57 MB 2025-02-14 10:21:17,062 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47636.81 MB 2025-02-14 10:21:17,062 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 10:21:17,062 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39300.91 MB 2025-02-14 10:21:17,081 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 10:21:17,081 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 10:21:17,081 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:21:17,081 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:21:17,081 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39006.01 MB 2025-02-14 10:21:17,081 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39235.44 MB 2025-02-14 10:21:17,081 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.43 MB 2025-02-14 10:21:17,081 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47636.81 MB 2025-02-14 10:21:17,081 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47636.81 MB 2025-02-14 10:21:17,081 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:21:17,081 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39433.85 MB 2025-02-14 10:21:17,082 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 10:21:17,082 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 10:21:17,082 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.39 seconds 2025-02-14 10:21:17,082 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:21:17,082 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26252.00 MB 2025-02-14 10:21:17,082 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39436.52 MB 2025-02-14 10:21:17,082 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13184.51 MB 2025-02-14 10:21:17,082 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53127.15 MB 2025-02-14 10:21:17,082 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47636.81 MB 2025-02-14 10:21:17,082 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5490.34 MB 2025-02-14 10:21:17,082 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39436.52 MB 2025-02-14 10:21:17,350 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 10:21:17,350 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 10:21:17,350 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 10:21:17,350 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:21:17,350 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39436.52 MB 2025-02-14 10:21:17,350 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31257.29 MB 2025-02-14 10:21:17,350 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8179.22 MB 2025-02-14 10:21:17,350 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47636.81 MB 2025-02-14 10:21:17,350 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47636.81 MB 2025-02-14 10:21:17,350 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:21:17,350 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41948.18 MB 2025-02-14 10:21:17,368 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 10:21:17,368 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 10:21:17,374 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 10:21:17,374 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 10:21:17,374 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:21:17,374 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:21:17,374 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31257.29 MB 2025-02-14 10:21:17,374 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39696.32 MB 2025-02-14 10:21:17,374 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 10:21:17,374 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47636.81 MB 2025-02-14 10:21:17,374 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56027.51 MB 2025-02-14 10:21:17,374 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 10:21:17,374 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39696.32 MB 2025-02-14 10:21:17,537 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 10:21:17,538 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:21:17,538 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 10:21:17,539 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:21:17,539 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 10:21:17,544 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 10:21:17,545 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:21:17,545 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 10:21:17,545 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 10:22:10,932 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:22:10,932 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 10:22:10,941 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 10:22:10,947 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:22:10,947 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1127, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 10:22:10,949 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:22:10,949 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1127, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 10:22:28,305 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 10:22:28,305 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 10:22:28,305 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.35 seconds 2025-02-14 10:22:28,305 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:22:28,305 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20821.82 MB 2025-02-14 10:22:28,305 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24810.54 MB 2025-02-14 10:22:28,305 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3988.72 MB 2025-02-14 10:22:28,305 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 68612.52 MB 2025-02-14 10:22:28,305 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32971.42 MB 2025-02-14 10:22:28,305 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35641.10 MB 2025-02-14 10:22:28,305 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33690.51 MB 2025-02-14 10:22:28,372 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 10:22:28,372 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 10:22:28,372 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 10:22:28,372 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:22:28,372 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24810.54 MB 2025-02-14 10:22:28,372 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21636.77 MB 2025-02-14 10:22:28,372 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3173.77 MB 2025-02-14 10:22:28,372 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32971.42 MB 2025-02-14 10:22:28,372 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40701.53 MB 2025-02-14 10:22:28,372 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7730.10 MB 2025-02-14 10:22:28,372 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36785.49 MB 2025-02-14 10:22:30,286 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 10:22:30,286 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 10:22:30,286 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 10:22:30,286 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:22:30,286 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21636.77 MB 2025-02-14 10:22:30,286 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22167.61 MB 2025-02-14 10:22:30,286 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 10:22:30,286 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40701.53 MB 2025-02-14 10:22:30,286 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25283.26 MB 2025-02-14 10:22:30,286 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15418.26 MB 2025-02-14 10:22:30,286 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26146.95 MB 2025-02-14 10:22:30,300 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 10:22:30,300 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 10:22:30,300 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:22:30,300 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:22:30,300 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22167.61 MB 2025-02-14 10:22:30,300 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24057.15 MB 2025-02-14 10:22:30,300 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 10:22:30,300 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25283.26 MB 2025-02-14 10:22:30,300 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28114.42 MB 2025-02-14 10:22:30,300 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-14 10:22:30,300 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25474.58 MB 2025-02-14 10:22:30,509 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 10:22:30,509 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 10:22:30,509 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 10:22:30,509 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:22:30,509 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24057.15 MB 2025-02-14 10:22:30,509 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26299.00 MB 2025-02-14 10:22:30,509 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 10:22:30,509 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28114.42 MB 2025-02-14 10:22:30,509 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34248.59 MB 2025-02-14 10:22:30,509 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 10:22:30,510 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31843.28 MB 2025-02-14 10:22:30,510 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 10:22:30,510 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 10:22:30,510 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 10:22:30,510 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:22:30,510 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22167.61 MB 2025-02-14 10:22:30,510 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26299.00 MB 2025-02-14 10:22:30,510 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 10:22:30,510 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25283.26 MB 2025-02-14 10:22:30,510 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34248.59 MB 2025-02-14 10:22:30,510 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8965.32 MB 2025-02-14 10:22:30,510 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31843.28 MB 2025-02-14 10:22:30,678 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 10:22:30,678 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 10:22:30,678 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 10:22:30,678 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:22:30,678 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27832.55 MB 2025-02-14 10:22:30,678 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28599.55 MB 2025-02-14 10:22:30,678 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 10:22:30,678 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34248.59 MB 2025-02-14 10:22:30,678 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34665.92 MB 2025-02-14 10:22:30,678 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 10:22:30,678 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29307.34 MB 2025-02-14 10:22:30,697 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 10:22:30,697 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 10:22:30,697 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:22:30,697 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:22:30,697 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29012.44 MB 2025-02-14 10:22:30,697 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29240.59 MB 2025-02-14 10:22:30,697 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.15 MB 2025-02-14 10:22:30,697 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34665.92 MB 2025-02-14 10:22:30,697 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34665.92 MB 2025-02-14 10:22:30,697 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:22:30,697 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29456.53 MB 2025-02-14 10:22:30,699 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 10:22:30,699 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 10:22:30,699 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.75 seconds 2025-02-14 10:22:30,699 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:22:30,699 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16895.26 MB 2025-02-14 10:22:30,699 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29441.66 MB 2025-02-14 10:22:30,699 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12546.40 MB 2025-02-14 10:22:30,699 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 68612.52 MB 2025-02-14 10:22:30,699 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34665.92 MB 2025-02-14 10:22:30,699 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33946.60 MB 2025-02-14 10:22:30,699 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29456.53 MB 2025-02-14 10:22:30,965 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 10:22:30,965 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 10:22:30,965 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 10:22:30,965 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:22:30,965 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18885.62 MB 2025-02-14 10:22:30,965 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21899.65 MB 2025-02-14 10:22:30,965 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-14 10:22:30,965 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34665.92 MB 2025-02-14 10:22:30,965 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34665.92 MB 2025-02-14 10:22:30,965 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:22:30,965 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22201.02 MB 2025-02-14 10:22:30,982 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 10:22:30,983 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 10:22:30,989 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 10:22:30,989 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 10:22:30,989 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:22:30,989 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:22:30,989 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21899.65 MB 2025-02-14 10:22:30,989 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30338.68 MB 2025-02-14 10:22:30,989 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 10:22:30,989 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34665.92 MB 2025-02-14 10:22:30,989 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43056.63 MB 2025-02-14 10:22:30,989 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 10:22:30,989 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30338.68 MB 2025-02-14 10:22:31,156 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 10:22:31,157 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:22:31,157 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 10:22:31,159 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:22:31,159 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 10:22:31,163 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 10:22:31,164 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:22:31,164 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 10:22:31,164 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 10:23:25,780 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:23:25,781 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 10:23:25,786 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 10:23:25,789 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:23:25,789 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1183, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 10:23:25,790 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:23:25,790 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1183, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 10:23:43,987 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 10:23:43,988 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 10:23:43,988 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.19 seconds 2025-02-14 10:23:43,988 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:23:43,988 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21212.04 MB 2025-02-14 10:23:43,988 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25398.61 MB 2025-02-14 10:23:43,988 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4186.57 MB 2025-02-14 10:23:43,988 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55641.64 MB 2025-02-14 10:23:43,988 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29477.57 MB 2025-02-14 10:23:43,988 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26164.07 MB 2025-02-14 10:23:43,988 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34308.10 MB 2025-02-14 10:23:44,081 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 10:23:44,081 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 10:23:44,081 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 10:23:44,081 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:23:44,081 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25398.61 MB 2025-02-14 10:23:44,081 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21927.90 MB 2025-02-14 10:23:44,081 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3470.71 MB 2025-02-14 10:23:44,081 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29477.57 MB 2025-02-14 10:23:44,081 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45229.28 MB 2025-02-14 10:23:44,081 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 15751.71 MB 2025-02-14 10:23:44,081 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37984.72 MB 2025-02-14 10:23:45,994 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 10:23:45,994 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 10:23:45,994 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 10:23:45,994 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:23:45,994 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21927.90 MB 2025-02-14 10:23:45,994 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22458.74 MB 2025-02-14 10:23:45,994 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 10:23:45,994 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45229.28 MB 2025-02-14 10:23:45,994 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26705.13 MB 2025-02-14 10:23:45,994 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18524.14 MB 2025-02-14 10:23:45,994 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26438.07 MB 2025-02-14 10:23:46,007 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 10:23:46,007 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 10:23:46,007 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:23:46,007 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:23:46,007 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22458.74 MB 2025-02-14 10:23:46,007 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24348.27 MB 2025-02-14 10:23:46,007 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 10:23:46,007 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26705.13 MB 2025-02-14 10:23:46,007 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28592.57 MB 2025-02-14 10:23:46,007 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 10:23:46,007 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25765.70 MB 2025-02-14 10:23:46,219 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 10:23:46,219 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 10:23:46,219 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 10:23:46,219 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:23:46,219 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24348.27 MB 2025-02-14 10:23:46,219 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26590.13 MB 2025-02-14 10:23:46,219 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 10:23:46,219 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28592.57 MB 2025-02-14 10:23:46,219 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34254.88 MB 2025-02-14 10:23:46,219 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 10:23:46,219 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32134.41 MB 2025-02-14 10:23:46,219 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 10:23:46,219 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 10:23:46,219 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 10:23:46,219 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:23:46,219 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22458.74 MB 2025-02-14 10:23:46,219 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26590.13 MB 2025-02-14 10:23:46,219 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 10:23:46,219 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26705.13 MB 2025-02-14 10:23:46,219 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34254.88 MB 2025-02-14 10:23:46,220 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-14 10:23:46,220 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32134.41 MB 2025-02-14 10:23:46,387 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 10:23:46,387 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 10:23:46,387 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 10:23:46,387 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:23:46,387 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28123.67 MB 2025-02-14 10:23:46,387 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28890.67 MB 2025-02-14 10:23:46,387 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 10:23:46,387 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34254.88 MB 2025-02-14 10:23:46,387 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34672.21 MB 2025-02-14 10:23:46,387 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 10:23:46,387 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29598.46 MB 2025-02-14 10:23:46,406 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 10:23:46,406 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 10:23:46,406 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:23:46,406 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:23:46,406 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29303.56 MB 2025-02-14 10:23:46,406 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29530.95 MB 2025-02-14 10:23:46,406 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.39 MB 2025-02-14 10:23:46,406 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34672.21 MB 2025-02-14 10:23:46,406 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34672.21 MB 2025-02-14 10:23:46,406 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:23:46,406 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29766.69 MB 2025-02-14 10:23:46,407 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 10:23:46,407 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 10:23:46,407 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.62 seconds 2025-02-14 10:23:46,407 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:23:46,407 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17090.37 MB 2025-02-14 10:23:46,407 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29731.09 MB 2025-02-14 10:23:46,407 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12640.72 MB 2025-02-14 10:23:46,407 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55641.64 MB 2025-02-14 10:23:46,407 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34672.21 MB 2025-02-14 10:23:46,407 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20969.42 MB 2025-02-14 10:23:46,407 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29766.69 MB 2025-02-14 10:23:46,673 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 10:23:46,673 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 10:23:46,673 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 10:23:46,673 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:23:46,673 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29731.09 MB 2025-02-14 10:23:46,673 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22080.29 MB 2025-02-14 10:23:46,673 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7650.80 MB 2025-02-14 10:23:46,673 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34672.21 MB 2025-02-14 10:23:46,673 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34672.21 MB 2025-02-14 10:23:46,673 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:23:46,673 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32231.08 MB 2025-02-14 10:23:46,691 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8124, cut from 8126 2025-02-14 10:23:46,691 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 10:23:46,697 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 10:23:46,697 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 10:23:46,697 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:23:46,697 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:23:46,697 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22080.29 MB 2025-02-14 10:23:46,697 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30481.15 MB 2025-02-14 10:23:46,697 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8400.86 MB 2025-02-14 10:23:46,697 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34672.21 MB 2025-02-14 10:23:46,697 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43023.07 MB 2025-02-14 10:23:46,697 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8350.86 MB 2025-02-14 10:23:46,697 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30481.15 MB 2025-02-14 10:23:46,859 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7916] 2025-02-14 10:23:46,860 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:23:46,860 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 10:23:46,861 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:23:46,861 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 10:23:46,866 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 10:23:46,867 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:23:46,867 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 10:23:46,867 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 10:24:01,073 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:24:01,074 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 10:24:01,079 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 10:24:01,082 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:24:01,082 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1257, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 10:24:01,083 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:24:01,083 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1257, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 10:24:20,492 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 10:24:20,492 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 10:24:20,492 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.40 seconds 2025-02-14 10:24:20,492 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:24:20,492 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21727.68 MB 2025-02-14 10:24:20,492 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26176.14 MB 2025-02-14 10:24:20,492 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4448.45 MB 2025-02-14 10:24:20,492 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51373.93 MB 2025-02-14 10:24:20,492 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38077.99 MB 2025-02-14 10:24:20,492 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13295.94 MB 2025-02-14 10:24:20,492 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35049.43 MB 2025-02-14 10:24:20,564 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 10:24:20,564 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 10:24:20,564 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 10:24:20,564 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:24:20,564 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26176.14 MB 2025-02-14 10:24:20,564 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22312.60 MB 2025-02-14 10:24:20,564 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3863.53 MB 2025-02-14 10:24:20,564 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38077.99 MB 2025-02-14 10:24:20,564 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46818.92 MB 2025-02-14 10:24:20,564 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8740.93 MB 2025-02-14 10:24:20,564 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39347.94 MB 2025-02-14 10:24:22,484 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 10:24:22,484 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 10:24:22,484 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 10:24:22,484 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:24:22,484 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22312.60 MB 2025-02-14 10:24:22,484 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22843.44 MB 2025-02-14 10:24:22,484 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 10:24:22,484 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46818.92 MB 2025-02-14 10:24:22,484 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33627.83 MB 2025-02-14 10:24:22,484 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13191.09 MB 2025-02-14 10:24:22,484 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26822.78 MB 2025-02-14 10:24:22,497 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 10:24:22,497 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 10:24:22,498 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:24:22,498 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:24:22,498 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22843.44 MB 2025-02-14 10:24:22,498 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24732.98 MB 2025-02-14 10:24:22,498 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 10:24:22,498 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33627.83 MB 2025-02-14 10:24:22,498 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33627.83 MB 2025-02-14 10:24:22,498 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:24:22,498 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26150.41 MB 2025-02-14 10:24:22,708 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 10:24:22,708 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 10:24:22,708 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 10:24:22,708 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:24:22,708 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24732.98 MB 2025-02-14 10:24:22,708 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26974.83 MB 2025-02-14 10:24:22,708 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 10:24:22,708 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33627.83 MB 2025-02-14 10:24:22,708 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35515.27 MB 2025-02-14 10:24:22,708 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 10:24:22,708 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32519.11 MB 2025-02-14 10:24:22,708 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 10:24:22,708 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 10:24:22,708 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 10:24:22,708 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:24:22,708 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22843.44 MB 2025-02-14 10:24:22,708 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26974.83 MB 2025-02-14 10:24:22,708 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 10:24:22,708 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33627.83 MB 2025-02-14 10:24:22,708 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35515.27 MB 2025-02-14 10:24:22,709 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 10:24:22,709 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32519.11 MB 2025-02-14 10:24:22,876 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 10:24:22,876 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 10:24:22,876 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 10:24:22,876 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:24:22,876 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28508.38 MB 2025-02-14 10:24:22,876 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29275.38 MB 2025-02-14 10:24:22,876 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 10:24:22,876 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35515.27 MB 2025-02-14 10:24:22,876 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35932.60 MB 2025-02-14 10:24:22,876 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 10:24:22,876 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29983.17 MB 2025-02-14 10:24:22,895 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 10:24:22,895 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 10:24:22,895 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:24:22,895 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:24:22,895 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29688.27 MB 2025-02-14 10:24:22,895 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29917.87 MB 2025-02-14 10:24:22,895 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.60 MB 2025-02-14 10:24:22,895 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35932.60 MB 2025-02-14 10:24:22,895 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35932.60 MB 2025-02-14 10:24:22,895 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:24:22,895 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30159.36 MB 2025-02-14 10:24:22,896 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 10:24:22,896 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 10:24:22,896 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.81 seconds 2025-02-14 10:24:22,896 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:24:22,896 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17348.19 MB 2025-02-14 10:24:22,896 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30118.55 MB 2025-02-14 10:24:22,896 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12770.35 MB 2025-02-14 10:24:22,896 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51373.93 MB 2025-02-14 10:24:22,896 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35932.60 MB 2025-02-14 10:24:22,896 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15441.33 MB 2025-02-14 10:24:22,896 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30159.36 MB 2025-02-14 10:24:23,180 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 10:24:23,180 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 10:24:23,180 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.28 seconds 2025-02-14 10:24:23,180 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:24:23,180 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30118.55 MB 2025-02-14 10:24:23,180 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22346.49 MB 2025-02-14 10:24:23,180 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7772.06 MB 2025-02-14 10:24:23,180 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35932.60 MB 2025-02-14 10:24:23,180 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35932.60 MB 2025-02-14 10:24:23,180 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:24:23,180 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32625.30 MB 2025-02-14 10:24:23,200 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8146, cut from 8148 2025-02-14 10:24:23,200 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 10:24:23,207 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 10:24:23,207 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 10:24:23,207 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:24:23,208 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:24:23,208 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22346.49 MB 2025-02-14 10:24:23,208 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30768.82 MB 2025-02-14 10:24:23,208 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8422.33 MB 2025-02-14 10:24:23,208 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35932.60 MB 2025-02-14 10:24:23,208 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44306.53 MB 2025-02-14 10:24:23,208 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8373.93 MB 2025-02-14 10:24:23,208 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30768.82 MB 2025-02-14 10:24:23,471 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7938] 2025-02-14 10:24:23,473 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:24:23,473 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 10:24:23,475 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:24:23,475 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 10:24:23,483 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 10:24:23,485 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:24:23,485 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 10:24:23,485 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 10:25:35,911 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:25:35,912 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 10:25:35,917 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 10:25:35,920 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:25:35,921 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 324, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 10:25:35,921 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:25:35,921 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 324, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 10:25:40,923 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 10:25:40,923 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 10:25:40,923 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.00 seconds 2025-02-14 10:25:40,923 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:25:40,923 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15226.39 MB 2025-02-14 10:25:40,923 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16373.53 MB 2025-02-14 10:25:40,923 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1147.14 MB 2025-02-14 10:25:40,923 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56866.37 MB 2025-02-14 10:25:40,923 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19000.20 MB 2025-02-14 10:25:40,923 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -37866.18 MB 2025-02-14 10:25:40,923 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25378.44 MB 2025-02-14 10:25:40,944 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 10:25:40,944 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 10:25:40,944 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:25:40,944 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:25:40,944 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16373.53 MB 2025-02-14 10:25:40,944 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16845.25 MB 2025-02-14 10:25:40,944 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 471.72 MB 2025-02-14 10:25:40,944 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19000.20 MB 2025-02-14 10:25:40,944 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22837.99 MB 2025-02-14 10:25:40,944 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3837.79 MB 2025-02-14 10:25:40,944 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20772.15 MB 2025-02-14 10:25:42,437 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 10:25:42,437 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 10:25:42,437 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.49 seconds 2025-02-14 10:25:42,437 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:25:42,437 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16845.25 MB 2025-02-14 10:25:42,437 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17259.30 MB 2025-02-14 10:25:42,437 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 414.06 MB 2025-02-14 10:25:42,437 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22837.99 MB 2025-02-14 10:25:42,437 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18551.41 MB 2025-02-14 10:25:42,437 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4286.58 MB 2025-02-14 10:25:42,437 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21187.63 MB 2025-02-14 10:25:42,449 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 10:25:42,449 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 10:25:42,449 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:25:42,449 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:25:42,449 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17259.30 MB 2025-02-14 10:25:42,449 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18733.42 MB 2025-02-14 10:25:42,449 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1474.11 MB 2025-02-14 10:25:42,449 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18551.41 MB 2025-02-14 10:25:42,449 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21128.81 MB 2025-02-14 10:25:42,449 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2577.40 MB 2025-02-14 10:25:42,449 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19840.06 MB 2025-02-14 10:25:42,616 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 10:25:42,616 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 10:25:42,616 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 10:25:42,616 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:25:42,616 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18733.42 MB 2025-02-14 10:25:42,616 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20482.08 MB 2025-02-14 10:25:42,616 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1748.66 MB 2025-02-14 10:25:42,616 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21128.81 MB 2025-02-14 10:25:42,616 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26283.61 MB 2025-02-14 10:25:42,616 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5154.80 MB 2025-02-14 10:25:42,616 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24809.75 MB 2025-02-14 10:25:42,617 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 10:25:42,617 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 10:25:42,617 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-14 10:25:42,617 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:25:42,617 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17259.30 MB 2025-02-14 10:25:42,617 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20482.08 MB 2025-02-14 10:25:42,617 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3222.78 MB 2025-02-14 10:25:42,617 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18551.41 MB 2025-02-14 10:25:42,617 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26283.61 MB 2025-02-14 10:25:42,617 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7732.20 MB 2025-02-14 10:25:42,617 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24809.75 MB 2025-02-14 10:25:42,748 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 10:25:42,748 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 10:25:42,748 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 10:25:42,748 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:25:42,748 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21678.24 MB 2025-02-14 10:25:42,748 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22278.60 MB 2025-02-14 10:25:42,748 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 600.36 MB 2025-02-14 10:25:42,748 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26283.61 MB 2025-02-14 10:25:42,748 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26606.57 MB 2025-02-14 10:25:42,748 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 322.96 MB 2025-02-14 10:25:42,748 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22830.68 MB 2025-02-14 10:25:42,763 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 10:25:42,763 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 10:25:42,763 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:25:42,763 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:25:42,763 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22600.66 MB 2025-02-14 10:25:42,763 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22829.45 MB 2025-02-14 10:25:42,763 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.79 MB 2025-02-14 10:25:42,763 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26606.57 MB 2025-02-14 10:25:42,763 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26606.57 MB 2025-02-14 10:25:42,763 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:25:42,763 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22965.53 MB 2025-02-14 10:25:42,765 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 10:25:42,765 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 10:25:42,765 - resource_logging.py:150 - __exit__ - DEBUG - Time: 6.84 seconds 2025-02-14 10:25:42,765 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:25:42,765 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14097.55 MB 2025-02-14 10:25:42,765 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23030.52 MB 2025-02-14 10:25:42,765 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8932.97 MB 2025-02-14 10:25:42,765 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56866.37 MB 2025-02-14 10:25:42,765 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26606.57 MB 2025-02-14 10:25:42,765 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30259.81 MB 2025-02-14 10:25:42,765 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23030.52 MB 2025-02-14 10:25:43,033 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 10:25:43,033 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 10:25:43,033 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 10:25:43,033 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:25:43,033 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23030.52 MB 2025-02-14 10:25:43,033 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26044.55 MB 2025-02-14 10:25:43,033 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-14 10:25:43,033 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26606.57 MB 2025-02-14 10:25:43,033 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27277.66 MB 2025-02-14 10:25:43,033 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 671.09 MB 2025-02-14 10:25:43,033 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26346.18 MB 2025-02-14 10:25:43,051 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 10:25:43,051 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 10:25:43,057 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 10:25:43,058 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 10:25:43,058 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:25:43,058 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:25:43,058 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18688.74 MB 2025-02-14 10:25:43,058 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27127.76 MB 2025-02-14 10:25:43,058 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 10:25:43,058 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27277.66 MB 2025-02-14 10:25:43,058 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37767.61 MB 2025-02-14 10:25:43,058 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 10:25:43,058 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27127.76 MB 2025-02-14 10:25:43,219 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 10:25:43,220 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:25:43,220 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 10:25:43,221 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:25:43,221 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 10:25:43,226 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 10:25:43,227 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:25:43,227 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 10:25:43,227 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 10:26:39,952 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:26:39,952 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 10:26:39,957 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 10:26:39,961 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:26:39,961 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1716, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 10:26:39,962 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:26:39,962 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1716, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 10:27:06,224 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 10:27:06,224 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 10:27:06,224 - resource_logging.py:150 - __exit__ - DEBUG - Time: 26.26 seconds 2025-02-14 10:27:06,224 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:27:06,224 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24926.07 MB 2025-02-14 10:27:06,224 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30999.42 MB 2025-02-14 10:27:06,224 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6073.35 MB 2025-02-14 10:27:06,224 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50352.62 MB 2025-02-14 10:27:06,224 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39764.10 MB 2025-02-14 10:27:06,224 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10588.52 MB 2025-02-14 10:27:06,224 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39834.63 MB 2025-02-14 10:27:06,323 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 10:27:06,323 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 10:27:06,323 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 10:27:06,323 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:27:06,323 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30999.42 MB 2025-02-14 10:27:06,323 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24698.80 MB 2025-02-14 10:27:06,323 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6300.62 MB 2025-02-14 10:27:06,323 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39764.10 MB 2025-02-14 10:27:06,323 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57204.02 MB 2025-02-14 10:27:06,323 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 17439.92 MB 2025-02-14 10:27:06,323 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48357.65 MB 2025-02-14 10:27:08,232 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 10:27:08,232 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 10:27:08,232 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 10:27:08,232 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:27:08,232 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24698.80 MB 2025-02-14 10:27:08,232 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25229.64 MB 2025-02-14 10:27:08,232 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 10:27:08,232 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57204.02 MB 2025-02-14 10:27:08,232 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30912.02 MB 2025-02-14 10:27:08,232 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26291.99 MB 2025-02-14 10:27:08,232 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29208.97 MB 2025-02-14 10:27:08,248 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 10:27:08,248 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 10:27:08,248 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:27:08,248 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:27:08,248 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25229.64 MB 2025-02-14 10:27:08,248 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27119.17 MB 2025-02-14 10:27:08,248 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 10:27:08,248 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30912.02 MB 2025-02-14 10:27:08,248 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31855.74 MB 2025-02-14 10:27:08,248 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 10:27:08,248 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28536.60 MB 2025-02-14 10:27:08,458 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 10:27:08,458 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 10:27:08,458 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 10:27:08,458 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:27:08,458 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27119.17 MB 2025-02-14 10:27:08,458 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29361.03 MB 2025-02-14 10:27:08,458 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 10:27:08,458 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31855.74 MB 2025-02-14 10:27:08,458 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37989.91 MB 2025-02-14 10:27:08,458 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 10:27:08,458 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34905.31 MB 2025-02-14 10:27:08,459 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 10:27:08,459 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 10:27:08,459 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 10:27:08,459 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:27:08,459 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25229.64 MB 2025-02-14 10:27:08,459 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29361.03 MB 2025-02-14 10:27:08,459 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 10:27:08,459 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30912.02 MB 2025-02-14 10:27:08,459 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37989.91 MB 2025-02-14 10:27:08,459 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7077.89 MB 2025-02-14 10:27:08,459 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34905.31 MB 2025-02-14 10:27:08,628 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 10:27:08,628 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 10:27:08,628 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 10:27:08,628 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:27:08,628 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30894.57 MB 2025-02-14 10:27:08,628 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31661.57 MB 2025-02-14 10:27:08,628 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 10:27:08,628 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37989.91 MB 2025-02-14 10:27:08,628 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38403.05 MB 2025-02-14 10:27:08,628 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 10:27:08,628 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32369.36 MB 2025-02-14 10:27:08,647 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 10:27:08,647 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 10:27:08,647 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:27:08,647 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:27:08,647 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32074.46 MB 2025-02-14 10:27:08,647 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32301.20 MB 2025-02-14 10:27:08,647 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.74 MB 2025-02-14 10:27:08,647 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38403.05 MB 2025-02-14 10:27:08,647 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38403.05 MB 2025-02-14 10:27:08,647 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:27:08,647 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32519.24 MB 2025-02-14 10:27:08,648 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 10:27:08,648 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 10:27:08,648 - resource_logging.py:150 - __exit__ - DEBUG - Time: 28.68 seconds 2025-02-14 10:27:08,648 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:27:08,648 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18947.39 MB 2025-02-14 10:27:08,648 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32501.61 MB 2025-02-14 10:27:08,648 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13554.22 MB 2025-02-14 10:27:08,648 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50352.62 MB 2025-02-14 10:27:08,648 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38403.05 MB 2025-02-14 10:27:08,648 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11949.57 MB 2025-02-14 10:27:08,648 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32519.24 MB 2025-02-14 10:27:08,916 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 10:27:08,916 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 10:27:08,916 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 10:27:08,916 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:27:08,916 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32501.61 MB 2025-02-14 10:27:08,916 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23941.49 MB 2025-02-14 10:27:08,916 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8560.12 MB 2025-02-14 10:27:08,916 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38403.05 MB 2025-02-14 10:27:08,916 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38403.05 MB 2025-02-14 10:27:08,916 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:27:08,916 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35004.98 MB 2025-02-14 10:27:08,943 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8135, cut from 8137 2025-02-14 10:27:08,944 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 10:27:08,958 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 10:27:08,958 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 10:27:08,958 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-14 10:27:08,958 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:27:08,958 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23941.49 MB 2025-02-14 10:27:08,958 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32352.31 MB 2025-02-14 10:27:08,958 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8410.82 MB 2025-02-14 10:27:08,958 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38403.05 MB 2025-02-14 10:27:08,958 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46766.49 MB 2025-02-14 10:27:08,958 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8363.44 MB 2025-02-14 10:27:08,958 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32352.31 MB 2025-02-14 10:27:09,122 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7927] 2025-02-14 10:27:09,123 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:27:09,123 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 10:27:09,124 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:27:09,124 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 10:27:09,129 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 10:27:09,130 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:27:09,130 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 10:27:09,130 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 10:27:18,100 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:27:18,100 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 10:27:18,105 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 10:27:18,108 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:27:18,108 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1468, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 10:27:18,109 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:27:18,109 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1468, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 10:27:40,756 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 10:27:40,756 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 10:27:40,756 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.64 seconds 2025-02-14 10:27:40,756 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:27:40,756 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23197.97 MB 2025-02-14 10:27:40,756 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28393.14 MB 2025-02-14 10:27:40,756 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5195.17 MB 2025-02-14 10:27:40,756 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55129.93 MB 2025-02-14 10:27:40,756 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38845.55 MB 2025-02-14 10:27:40,756 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16284.39 MB 2025-02-14 10:27:40,757 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37199.18 MB 2025-02-14 10:27:40,838 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 10:27:40,838 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 10:27:40,838 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 10:27:40,838 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:27:40,838 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28393.14 MB 2025-02-14 10:27:40,838 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23409.52 MB 2025-02-14 10:27:40,838 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4983.61 MB 2025-02-14 10:27:40,838 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38845.55 MB 2025-02-14 10:27:40,838 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48651.83 MB 2025-02-14 10:27:40,838 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9806.28 MB 2025-02-14 10:27:40,838 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43181.96 MB 2025-02-14 10:27:42,761 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 10:27:42,761 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 10:27:42,761 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 10:27:42,761 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:27:42,761 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23409.52 MB 2025-02-14 10:27:42,761 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23940.37 MB 2025-02-14 10:27:42,761 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 10:27:42,761 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48651.83 MB 2025-02-14 10:27:42,761 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29467.08 MB 2025-02-14 10:27:42,761 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19184.75 MB 2025-02-14 10:27:42,761 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27919.70 MB 2025-02-14 10:27:42,776 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 10:27:42,776 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 10:27:42,777 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:27:42,777 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:27:42,777 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23940.37 MB 2025-02-14 10:27:42,777 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25829.90 MB 2025-02-14 10:27:42,777 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 10:27:42,777 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29467.08 MB 2025-02-14 10:27:42,777 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30410.80 MB 2025-02-14 10:27:42,777 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 10:27:42,777 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27247.33 MB 2025-02-14 10:27:42,985 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 10:27:42,985 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 10:27:42,985 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 10:27:42,985 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:27:42,985 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25829.90 MB 2025-02-14 10:27:42,985 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28071.76 MB 2025-02-14 10:27:42,985 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 10:27:42,985 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30410.80 MB 2025-02-14 10:27:42,985 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36073.11 MB 2025-02-14 10:27:42,985 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 10:27:42,985 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33616.04 MB 2025-02-14 10:27:42,986 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 10:27:42,986 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 10:27:42,986 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 10:27:42,986 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:27:42,986 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23940.37 MB 2025-02-14 10:27:42,986 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28071.76 MB 2025-02-14 10:27:42,986 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 10:27:42,986 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29467.08 MB 2025-02-14 10:27:42,986 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36073.11 MB 2025-02-14 10:27:42,986 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 10:27:42,986 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33616.04 MB 2025-02-14 10:27:43,152 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 10:27:43,152 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 10:27:43,152 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 10:27:43,152 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:27:43,152 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29605.30 MB 2025-02-14 10:27:43,152 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30372.30 MB 2025-02-14 10:27:43,152 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 10:27:43,152 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36073.11 MB 2025-02-14 10:27:43,152 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36490.44 MB 2025-02-14 10:27:43,152 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 10:27:43,152 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31080.09 MB 2025-02-14 10:27:43,171 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 10:27:43,171 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 10:27:43,171 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:27:43,171 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:27:43,171 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30785.19 MB 2025-02-14 10:27:43,171 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31013.86 MB 2025-02-14 10:27:43,171 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.67 MB 2025-02-14 10:27:43,171 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36490.44 MB 2025-02-14 10:27:43,171 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36490.44 MB 2025-02-14 10:27:43,171 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:27:43,171 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31206.88 MB 2025-02-14 10:27:43,172 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 10:27:43,172 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 10:27:43,172 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.06 seconds 2025-02-14 10:27:43,172 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:27:43,172 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18083.34 MB 2025-02-14 10:27:43,172 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31214.44 MB 2025-02-14 10:27:43,172 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13131.10 MB 2025-02-14 10:27:43,172 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55129.93 MB 2025-02-14 10:27:43,172 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36490.44 MB 2025-02-14 10:27:43,172 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18639.49 MB 2025-02-14 10:27:43,172 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31214.44 MB 2025-02-14 10:27:43,440 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 10:27:43,440 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 10:27:43,440 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 10:27:43,440 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:27:43,440 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31214.44 MB 2025-02-14 10:27:43,440 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23080.11 MB 2025-02-14 10:27:43,440 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8134.33 MB 2025-02-14 10:27:43,440 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36490.44 MB 2025-02-14 10:27:43,440 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36490.44 MB 2025-02-14 10:27:43,440 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:27:43,441 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33719.96 MB 2025-02-14 10:27:43,458 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8142, cut from 8144 2025-02-14 10:27:43,458 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 10:27:43,464 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 10:27:43,464 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 10:27:43,464 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:27:43,464 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:27:43,464 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23080.11 MB 2025-02-14 10:27:43,464 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31498.26 MB 2025-02-14 10:27:43,465 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8418.15 MB 2025-02-14 10:27:43,465 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36490.44 MB 2025-02-14 10:27:43,465 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44860.18 MB 2025-02-14 10:27:43,465 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8369.73 MB 2025-02-14 10:27:43,465 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31498.26 MB 2025-02-14 10:27:43,630 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7934] 2025-02-14 10:27:43,631 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:27:43,631 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 10:27:43,632 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:27:43,632 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 10:27:43,637 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 10:27:43,638 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:27:43,638 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 10:27:43,638 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 10:28:53,232 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:28:53,233 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 10:28:53,240 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 10:28:53,245 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:28:53,245 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 153, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 10:28:53,247 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:28:53,247 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 153, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 10:28:55,677 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 10:28:55,677 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 10:28:55,677 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.42 seconds 2025-02-14 10:28:55,677 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:28:55,677 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14034.84 MB 2025-02-14 10:28:55,677 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14576.29 MB 2025-02-14 10:28:55,677 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 541.46 MB 2025-02-14 10:28:55,677 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57413.73 MB 2025-02-14 10:28:55,677 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18794.68 MB 2025-02-14 10:28:55,677 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -38619.05 MB 2025-02-14 10:28:55,677 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23507.01 MB 2025-02-14 10:28:55,691 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 10:28:55,691 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 10:28:55,691 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:28:55,691 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:28:55,691 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14576.29 MB 2025-02-14 10:28:55,691 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14803.51 MB 2025-02-14 10:28:55,691 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.22 MB 2025-02-14 10:28:55,691 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18794.68 MB 2025-02-14 10:28:55,691 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18794.68 MB 2025-02-14 10:28:55,691 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:28:55,691 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16687.02 MB 2025-02-14 10:28:56,426 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 10:28:56,426 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 10:28:56,426 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.73 seconds 2025-02-14 10:28:56,426 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:28:56,426 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14803.51 MB 2025-02-14 10:28:56,426 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14999.93 MB 2025-02-14 10:28:56,426 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 196.41 MB 2025-02-14 10:28:56,426 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18794.68 MB 2025-02-14 10:28:56,426 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18794.68 MB 2025-02-14 10:28:56,426 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:28:56,426 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18974.99 MB 2025-02-14 10:28:56,436 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 10:28:56,436 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 10:28:56,436 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 10:28:56,436 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:28:56,436 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14999.86 MB 2025-02-14 10:28:56,436 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15698.82 MB 2025-02-14 10:28:56,436 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 698.96 MB 2025-02-14 10:28:56,436 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18794.68 MB 2025-02-14 10:28:56,436 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18794.68 MB 2025-02-14 10:28:56,436 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:28:56,436 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16223.27 MB 2025-02-14 10:28:56,535 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 10:28:56,535 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 10:28:56,535 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 10:28:56,535 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:28:56,535 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15698.82 MB 2025-02-14 10:28:56,535 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16528.35 MB 2025-02-14 10:28:56,535 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 829.53 MB 2025-02-14 10:28:56,535 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18794.68 MB 2025-02-14 10:28:56,535 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19495.12 MB 2025-02-14 10:28:56,535 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 700.45 MB 2025-02-14 10:28:56,535 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18581.79 MB 2025-02-14 10:28:56,537 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 10:28:56,537 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 10:28:56,537 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 10:28:56,537 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:28:56,537 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14999.86 MB 2025-02-14 10:28:56,537 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16528.35 MB 2025-02-14 10:28:56,537 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1528.49 MB 2025-02-14 10:28:56,537 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18794.68 MB 2025-02-14 10:28:56,537 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19495.12 MB 2025-02-14 10:28:56,537 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 700.45 MB 2025-02-14 10:28:56,537 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18581.79 MB 2025-02-14 10:28:56,629 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 10:28:56,629 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 10:28:56,629 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 10:28:56,629 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:28:56,629 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17095.76 MB 2025-02-14 10:28:56,629 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17379.55 MB 2025-02-14 10:28:56,629 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 283.79 MB 2025-02-14 10:28:56,629 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19495.12 MB 2025-02-14 10:28:56,629 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19648.22 MB 2025-02-14 10:28:56,629 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 153.09 MB 2025-02-14 10:28:56,630 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17651.01 MB 2025-02-14 10:28:56,642 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 10:28:56,643 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 10:28:56,643 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:28:56,643 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:28:56,643 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17532.32 MB 2025-02-14 10:28:56,643 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17744.77 MB 2025-02-14 10:28:56,643 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 212.44 MB 2025-02-14 10:28:56,643 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19648.22 MB 2025-02-14 10:28:56,643 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19648.22 MB 2025-02-14 10:28:56,643 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:28:56,643 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17747.87 MB 2025-02-14 10:28:56,645 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 10:28:56,645 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 10:28:56,645 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.39 seconds 2025-02-14 10:28:56,645 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:28:56,645 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13501.77 MB 2025-02-14 10:28:56,645 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17945.42 MB 2025-02-14 10:28:56,645 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4443.65 MB 2025-02-14 10:28:56,645 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57413.73 MB 2025-02-14 10:28:56,645 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19648.22 MB 2025-02-14 10:28:56,645 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -37765.51 MB 2025-02-14 10:28:56,645 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17945.42 MB 2025-02-14 10:28:56,924 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 10:28:56,924 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 10:28:56,924 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.28 seconds 2025-02-14 10:28:56,924 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:28:56,924 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17945.42 MB 2025-02-14 10:28:56,924 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17310.42 MB 2025-02-14 10:28:56,924 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -635.00 MB 2025-02-14 10:28:56,924 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19648.22 MB 2025-02-14 10:28:56,924 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19916.65 MB 2025-02-14 10:28:56,924 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 268.44 MB 2025-02-14 10:28:56,924 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19048.80 MB 2025-02-14 10:28:56,943 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8145, cut from 8147 2025-02-14 10:28:56,943 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2,'] 2025-02-14 10:28:56,950 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 10:28:56,950 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 10:28:56,950 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:28:56,950 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:28:56,950 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17310.42 MB 2025-02-14 10:28:56,950 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25732.38 MB 2025-02-14 10:28:56,951 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8421.96 MB 2025-02-14 10:28:56,951 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19916.65 MB 2025-02-14 10:28:56,951 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30381.44 MB 2025-02-14 10:28:56,951 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10464.79 MB 2025-02-14 10:28:56,951 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25732.38 MB 2025-02-14 10:28:57,175 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7937] 2025-02-14 10:28:57,177 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:28:57,177 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 10:28:57,179 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:28:57,179 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 10:28:57,186 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 10:28:57,188 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:28:57,188 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 10:28:57,188 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2,'] 2025-02-14 10:30:10,216 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:30:10,216 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 10:30:10,221 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 10:30:10,225 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:30:10,225 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1572, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 10:30:10,226 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:30:10,226 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1572, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 10:30:34,319 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 10:30:34,319 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 10:30:34,319 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.09 seconds 2025-02-14 10:30:34,319 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:30:34,319 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23922.65 MB 2025-02-14 10:30:34,319 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29486.40 MB 2025-02-14 10:30:34,319 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5563.74 MB 2025-02-14 10:30:34,319 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38753.27 MB 2025-02-14 10:30:34,319 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39214.65 MB 2025-02-14 10:30:34,319 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 461.37 MB 2025-02-14 10:30:34,319 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38377.67 MB 2025-02-14 10:30:34,408 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 10:30:34,408 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 10:30:34,408 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 10:30:34,408 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:30:34,409 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29486.40 MB 2025-02-14 10:30:34,409 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23950.19 MB 2025-02-14 10:30:34,409 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5536.21 MB 2025-02-14 10:30:34,409 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39214.65 MB 2025-02-14 10:30:34,409 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50205.82 MB 2025-02-14 10:30:34,409 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10991.17 MB 2025-02-14 10:30:34,409 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44922.54 MB 2025-02-14 10:30:36,316 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 10:30:36,316 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 10:30:36,316 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 10:30:36,316 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:30:36,316 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23950.19 MB 2025-02-14 10:30:36,316 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24481.03 MB 2025-02-14 10:30:36,316 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 10:30:36,316 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50205.82 MB 2025-02-14 10:30:36,316 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29464.99 MB 2025-02-14 10:30:36,316 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20740.83 MB 2025-02-14 10:30:36,316 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28460.36 MB 2025-02-14 10:30:36,332 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 10:30:36,332 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 10:30:36,332 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:30:36,332 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:30:36,332 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24481.03 MB 2025-02-14 10:30:36,332 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26370.56 MB 2025-02-14 10:30:36,332 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 10:30:36,333 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29464.99 MB 2025-02-14 10:30:36,333 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30408.70 MB 2025-02-14 10:30:36,333 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 10:30:36,333 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27787.99 MB 2025-02-14 10:30:36,546 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 10:30:36,546 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 10:30:36,546 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 10:30:36,546 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:30:36,546 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26370.56 MB 2025-02-14 10:30:36,546 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28612.42 MB 2025-02-14 10:30:36,546 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 10:30:36,546 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30408.70 MB 2025-02-14 10:30:36,546 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36542.87 MB 2025-02-14 10:30:36,546 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 10:30:36,546 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34156.70 MB 2025-02-14 10:30:36,546 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 10:30:36,546 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 10:30:36,546 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 10:30:36,546 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:30:36,546 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24481.03 MB 2025-02-14 10:30:36,547 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28612.42 MB 2025-02-14 10:30:36,547 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 10:30:36,547 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29464.99 MB 2025-02-14 10:30:36,547 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36542.87 MB 2025-02-14 10:30:36,547 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7077.89 MB 2025-02-14 10:30:36,547 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34156.70 MB 2025-02-14 10:30:36,721 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 10:30:36,721 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 10:30:36,721 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 10:30:36,722 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:30:36,722 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30145.96 MB 2025-02-14 10:30:36,722 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30912.96 MB 2025-02-14 10:30:36,722 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 10:30:36,722 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36542.87 MB 2025-02-14 10:30:36,722 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36960.21 MB 2025-02-14 10:30:36,722 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 10:30:36,722 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31620.75 MB 2025-02-14 10:30:36,741 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 10:30:36,741 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 10:30:36,741 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:30:36,741 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:30:36,741 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31325.85 MB 2025-02-14 10:30:36,741 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31553.99 MB 2025-02-14 10:30:36,741 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.14 MB 2025-02-14 10:30:36,741 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36960.21 MB 2025-02-14 10:30:36,741 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36960.21 MB 2025-02-14 10:30:36,741 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:30:36,741 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31763.13 MB 2025-02-14 10:30:36,742 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 10:30:36,742 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 10:30:36,742 - resource_logging.py:150 - __exit__ - DEBUG - Time: 26.51 seconds 2025-02-14 10:30:36,742 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:30:36,742 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18445.68 MB 2025-02-14 10:30:36,742 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31755.06 MB 2025-02-14 10:30:36,742 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13309.38 MB 2025-02-14 10:30:36,742 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38753.27 MB 2025-02-14 10:30:36,742 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36960.21 MB 2025-02-14 10:30:36,742 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1793.06 MB 2025-02-14 10:30:36,742 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31763.13 MB 2025-02-14 10:30:37,010 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 10:30:37,010 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 10:30:37,010 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 10:30:37,010 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:30:37,010 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31755.06 MB 2025-02-14 10:30:37,010 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23450.07 MB 2025-02-14 10:30:37,010 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8304.99 MB 2025-02-14 10:30:37,010 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36960.21 MB 2025-02-14 10:30:37,010 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36960.21 MB 2025-02-14 10:30:37,010 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:30:37,010 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34266.73 MB 2025-02-14 10:30:37,028 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 10:30:37,029 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 10:30:37,035 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 10:30:37,035 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 10:30:37,035 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:30:37,035 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:30:37,035 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23450.07 MB 2025-02-14 10:30:37,035 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31889.09 MB 2025-02-14 10:30:37,035 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 10:30:37,035 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36960.21 MB 2025-02-14 10:30:37,035 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45350.91 MB 2025-02-14 10:30:37,035 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 10:30:37,035 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31889.09 MB 2025-02-14 10:30:37,204 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 10:30:37,205 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:30:37,205 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 10:30:37,206 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:30:37,206 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 10:30:37,211 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 10:30:37,212 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:30:37,212 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 10:30:37,212 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 10:30:52,704 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:30:52,705 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 10:30:52,709 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 10:30:52,713 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:30:52,713 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1666, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 10:30:52,714 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:30:52,714 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1666, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 10:31:18,489 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 10:31:18,489 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 10:31:18,489 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.77 seconds 2025-02-14 10:31:18,489 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:31:18,489 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24577.66 MB 2025-02-14 10:31:18,489 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30473.54 MB 2025-02-14 10:31:18,489 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5895.88 MB 2025-02-14 10:31:18,489 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57935.92 MB 2025-02-14 10:31:18,489 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39558.58 MB 2025-02-14 10:31:18,489 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18377.34 MB 2025-02-14 10:31:18,489 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39485.66 MB 2025-02-14 10:31:18,579 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 10:31:18,579 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 10:31:18,579 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 10:31:18,579 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:31:18,579 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30473.54 MB 2025-02-14 10:31:18,579 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24438.86 MB 2025-02-14 10:31:18,579 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6034.68 MB 2025-02-14 10:31:18,579 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39558.58 MB 2025-02-14 10:31:18,579 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54406.41 MB 2025-02-14 10:31:18,579 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 14847.84 MB 2025-02-14 10:31:18,579 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46211.43 MB 2025-02-14 10:31:20,510 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 10:31:20,510 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 10:31:20,510 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 10:31:20,510 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:31:20,510 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24438.86 MB 2025-02-14 10:31:20,510 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24969.71 MB 2025-02-14 10:31:20,510 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 10:31:20,510 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54406.41 MB 2025-02-14 10:31:20,510 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35076.96 MB 2025-02-14 10:31:20,510 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19329.45 MB 2025-02-14 10:31:20,510 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28949.04 MB 2025-02-14 10:31:20,523 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 10:31:20,523 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 10:31:20,523 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:31:20,523 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:31:20,523 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24969.71 MB 2025-02-14 10:31:20,523 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26859.24 MB 2025-02-14 10:31:20,523 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 10:31:20,523 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35076.96 MB 2025-02-14 10:31:20,523 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35076.96 MB 2025-02-14 10:31:20,523 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:31:20,523 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28276.67 MB 2025-02-14 10:31:20,736 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 10:31:20,736 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 10:31:20,736 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 10:31:20,736 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:31:20,736 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26859.24 MB 2025-02-14 10:31:20,736 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29101.10 MB 2025-02-14 10:31:20,736 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 10:31:20,736 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35076.96 MB 2025-02-14 10:31:20,736 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37908.12 MB 2025-02-14 10:31:20,736 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-14 10:31:20,736 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34645.38 MB 2025-02-14 10:31:20,737 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 10:31:20,737 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 10:31:20,737 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 10:31:20,737 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:31:20,737 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24969.71 MB 2025-02-14 10:31:20,737 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29101.10 MB 2025-02-14 10:31:20,737 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 10:31:20,737 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35076.96 MB 2025-02-14 10:31:20,737 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37908.12 MB 2025-02-14 10:31:20,737 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-14 10:31:20,737 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34645.38 MB 2025-02-14 10:31:20,905 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 10:31:20,905 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 10:31:20,905 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 10:31:20,905 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:31:20,905 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30634.64 MB 2025-02-14 10:31:20,905 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31401.64 MB 2025-02-14 10:31:20,905 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 10:31:20,905 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37908.12 MB 2025-02-14 10:31:20,905 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38323.36 MB 2025-02-14 10:31:20,905 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 10:31:20,905 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32109.43 MB 2025-02-14 10:31:20,923 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 10:31:20,923 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 10:31:20,924 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:31:20,924 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:31:20,924 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31814.53 MB 2025-02-14 10:31:20,924 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32043.82 MB 2025-02-14 10:31:20,924 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.29 MB 2025-02-14 10:31:20,924 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38323.36 MB 2025-02-14 10:31:20,924 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38323.36 MB 2025-02-14 10:31:20,924 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:31:20,924 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32256.90 MB 2025-02-14 10:31:20,925 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 10:31:20,925 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 10:31:20,925 - resource_logging.py:150 - __exit__ - DEBUG - Time: 28.21 seconds 2025-02-14 10:31:20,925 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:31:20,925 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18773.18 MB 2025-02-14 10:31:20,925 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32244.90 MB 2025-02-14 10:31:20,925 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13471.71 MB 2025-02-14 10:31:20,925 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57935.92 MB 2025-02-14 10:31:20,925 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38323.36 MB 2025-02-14 10:31:20,925 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19612.57 MB 2025-02-14 10:31:20,925 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32256.90 MB 2025-02-14 10:31:21,194 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 10:31:21,194 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 10:31:21,194 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 10:31:21,194 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:31:21,194 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32244.90 MB 2025-02-14 10:31:21,194 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23777.57 MB 2025-02-14 10:31:21,194 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8467.32 MB 2025-02-14 10:31:21,194 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38323.36 MB 2025-02-14 10:31:21,194 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38323.36 MB 2025-02-14 10:31:21,194 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:31:21,194 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34756.56 MB 2025-02-14 10:31:21,212 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 10:31:21,212 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 10:31:21,218 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 10:31:21,218 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 10:31:21,218 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:31:21,218 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:31:21,218 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23777.57 MB 2025-02-14 10:31:21,218 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32216.60 MB 2025-02-14 10:31:21,218 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 10:31:21,218 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38323.36 MB 2025-02-14 10:31:21,218 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46714.06 MB 2025-02-14 10:31:21,218 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 10:31:21,218 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32216.60 MB 2025-02-14 10:31:21,381 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 10:31:21,383 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:31:21,383 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 10:31:21,384 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:31:21,384 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 10:31:21,389 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 10:31:21,390 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:31:21,390 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 10:31:21,390 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 10:32:53,858 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:32:53,858 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 10:32:53,863 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 10:32:53,867 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:32:53,867 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 352, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 10:32:53,868 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:32:53,868 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 352, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 10:32:59,259 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 10:32:59,259 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 10:32:59,259 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.39 seconds 2025-02-14 10:32:59,259 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:32:59,259 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15421.50 MB 2025-02-14 10:32:59,259 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16667.21 MB 2025-02-14 10:32:59,259 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1245.71 MB 2025-02-14 10:32:59,259 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59299.07 MB 2025-02-14 10:32:59,259 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20040.38 MB 2025-02-14 10:32:59,259 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -39258.69 MB 2025-02-14 10:32:59,259 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25573.15 MB 2025-02-14 10:32:59,283 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 10:32:59,283 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 10:32:59,283 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:32:59,283 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:32:59,283 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16667.21 MB 2025-02-14 10:32:59,283 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17271.12 MB 2025-02-14 10:32:59,283 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 603.91 MB 2025-02-14 10:32:59,283 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20040.38 MB 2025-02-14 10:32:59,283 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23739.76 MB 2025-02-14 10:32:59,283 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3699.38 MB 2025-02-14 10:32:59,283 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21669.70 MB 2025-02-14 10:33:00,945 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 10:33:00,945 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 10:33:00,945 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.66 seconds 2025-02-14 10:33:00,945 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:33:00,945 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17271.12 MB 2025-02-14 10:33:00,945 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17738.26 MB 2025-02-14 10:33:00,945 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 467.14 MB 2025-02-14 10:33:00,945 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23739.76 MB 2025-02-14 10:33:00,945 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20644.36 MB 2025-02-14 10:33:00,945 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3095.40 MB 2025-02-14 10:33:00,945 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21697.40 MB 2025-02-14 10:33:00,958 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 10:33:00,958 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 10:33:00,958 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:33:00,958 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:33:00,958 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17738.26 MB 2025-02-14 10:33:00,958 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19401.30 MB 2025-02-14 10:33:00,958 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1663.04 MB 2025-02-14 10:33:00,958 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20644.36 MB 2025-02-14 10:33:00,958 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23135.78 MB 2025-02-14 10:33:00,958 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2491.42 MB 2025-02-14 10:33:00,958 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20648.64 MB 2025-02-14 10:33:01,138 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 10:33:01,139 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 10:33:01,139 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-14 10:33:01,139 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:33:01,139 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19401.30 MB 2025-02-14 10:33:01,139 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21374.14 MB 2025-02-14 10:33:01,139 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1972.84 MB 2025-02-14 10:33:01,139 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23135.78 MB 2025-02-14 10:33:01,139 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28533.85 MB 2025-02-14 10:33:01,139 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5398.07 MB 2025-02-14 10:33:01,139 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26253.10 MB 2025-02-14 10:33:01,139 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 10:33:01,139 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 10:33:01,139 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.19 seconds 2025-02-14 10:33:01,139 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:33:01,139 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17738.26 MB 2025-02-14 10:33:01,139 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21374.14 MB 2025-02-14 10:33:01,139 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3635.88 MB 2025-02-14 10:33:01,139 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20644.36 MB 2025-02-14 10:33:01,139 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28533.85 MB 2025-02-14 10:33:01,139 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7889.49 MB 2025-02-14 10:33:01,139 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26253.10 MB 2025-02-14 10:33:01,281 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 10:33:01,281 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 10:33:01,281 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-14 10:33:01,281 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:33:01,281 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22723.66 MB 2025-02-14 10:33:01,281 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23398.62 MB 2025-02-14 10:33:01,281 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 674.96 MB 2025-02-14 10:33:01,281 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28533.85 MB 2025-02-14 10:33:01,281 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28900.85 MB 2025-02-14 10:33:01,281 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 367.00 MB 2025-02-14 10:33:01,281 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24021.48 MB 2025-02-14 10:33:01,298 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 10:33:01,298 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 10:33:01,298 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:33:01,298 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:33:01,298 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23761.97 MB 2025-02-14 10:33:01,298 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23987.87 MB 2025-02-14 10:33:01,298 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 225.91 MB 2025-02-14 10:33:01,298 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28900.85 MB 2025-02-14 10:33:01,298 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28900.85 MB 2025-02-14 10:33:01,298 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:33:01,298 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24165.14 MB 2025-02-14 10:33:01,299 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 10:33:01,299 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 10:33:01,299 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.43 seconds 2025-02-14 10:33:01,299 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:33:01,299 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14195.10 MB 2025-02-14 10:33:01,299 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24188.53 MB 2025-02-14 10:33:01,299 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 9993.43 MB 2025-02-14 10:33:01,299 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59299.07 MB 2025-02-14 10:33:01,299 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28900.85 MB 2025-02-14 10:33:01,299 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30398.22 MB 2025-02-14 10:33:01,299 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24188.53 MB 2025-02-14 10:33:01,565 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 10:33:01,565 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 10:33:01,565 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 10:33:01,565 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:33:01,565 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24188.53 MB 2025-02-14 10:33:01,565 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18966.49 MB 2025-02-14 10:33:01,565 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5222.04 MB 2025-02-14 10:33:01,565 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28900.85 MB 2025-02-14 10:33:01,565 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28900.85 MB 2025-02-14 10:33:01,565 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:33:01,565 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27396.78 MB 2025-02-14 10:33:01,583 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8145, cut from 8147 2025-02-14 10:33:01,584 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 10:33:01,590 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 10:33:01,590 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 10:33:01,590 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:33:01,590 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:33:01,590 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18966.49 MB 2025-02-14 10:33:01,590 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27388.45 MB 2025-02-14 10:33:01,590 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8421.96 MB 2025-02-14 10:33:01,590 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28900.85 MB 2025-02-14 10:33:01,590 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39365.64 MB 2025-02-14 10:33:01,590 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10464.79 MB 2025-02-14 10:33:01,590 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27388.45 MB 2025-02-14 10:33:01,741 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7937] 2025-02-14 10:33:01,743 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:33:01,743 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 10:33:01,744 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:33:01,744 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 10:33:01,748 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 10:33:01,749 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:33:01,749 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 10:33:01,750 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 10:33:10,990 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:33:10,990 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 10:33:10,994 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 10:33:10,998 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:33:10,998 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1990, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 10:33:10,999 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:33:10,999 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1990, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 10:33:41,724 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 10:33:41,724 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 10:33:41,724 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.72 seconds 2025-02-14 10:33:41,725 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:33:41,725 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26835.35 MB 2025-02-14 10:33:41,725 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33877.84 MB 2025-02-14 10:33:41,725 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7042.50 MB 2025-02-14 10:33:41,725 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47737.47 MB 2025-02-14 10:33:41,725 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40697.33 MB 2025-02-14 10:33:41,725 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7040.14 MB 2025-02-14 10:33:41,725 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42875.80 MB 2025-02-14 10:33:41,845 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 10:33:41,845 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 10:33:41,845 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 10:33:41,845 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:33:41,845 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33877.84 MB 2025-02-14 10:33:41,845 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26123.24 MB 2025-02-14 10:33:41,845 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7754.60 MB 2025-02-14 10:33:41,845 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40697.33 MB 2025-02-14 10:33:41,845 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 63294.14 MB 2025-02-14 10:33:41,845 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 22596.81 MB 2025-02-14 10:33:41,845 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 53652.33 MB 2025-02-14 10:33:43,807 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 10:33:43,807 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 10:33:43,807 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.96 seconds 2025-02-14 10:33:43,807 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:33:43,807 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26123.24 MB 2025-02-14 10:33:43,807 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26654.08 MB 2025-02-14 10:33:43,807 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 10:33:43,807 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63294.14 MB 2025-02-14 10:33:43,807 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30882.66 MB 2025-02-14 10:33:43,807 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32411.48 MB 2025-02-14 10:33:43,807 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30634.45 MB 2025-02-14 10:33:43,820 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 10:33:43,820 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 10:33:43,820 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:33:43,820 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:33:43,820 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26654.08 MB 2025-02-14 10:33:43,820 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28543.62 MB 2025-02-14 10:33:43,820 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 10:33:43,820 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30882.66 MB 2025-02-14 10:33:43,820 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32770.10 MB 2025-02-14 10:33:43,821 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 10:33:43,821 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29961.04 MB 2025-02-14 10:33:44,033 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 10:33:44,033 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 10:33:44,033 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 10:33:44,033 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:33:44,033 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28543.62 MB 2025-02-14 10:33:44,033 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30785.47 MB 2025-02-14 10:33:44,033 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 10:33:44,033 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32770.10 MB 2025-02-14 10:33:44,033 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38904.27 MB 2025-02-14 10:33:44,033 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 10:33:44,033 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36329.75 MB 2025-02-14 10:33:44,034 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 10:33:44,034 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 10:33:44,034 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 10:33:44,034 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:33:44,034 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26654.08 MB 2025-02-14 10:33:44,034 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30785.47 MB 2025-02-14 10:33:44,034 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 10:33:44,034 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30882.66 MB 2025-02-14 10:33:44,034 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38904.27 MB 2025-02-14 10:33:44,034 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-14 10:33:44,034 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36329.75 MB 2025-02-14 10:33:44,208 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 10:33:44,208 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 10:33:44,208 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 10:33:44,208 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:33:44,208 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32319.01 MB 2025-02-14 10:33:44,208 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33086.02 MB 2025-02-14 10:33:44,208 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 10:33:44,208 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38904.27 MB 2025-02-14 10:33:44,208 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39319.50 MB 2025-02-14 10:33:44,208 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 10:33:44,208 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33793.80 MB 2025-02-14 10:33:44,227 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 10:33:44,228 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 10:33:44,228 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:33:44,228 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:33:44,228 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33498.91 MB 2025-02-14 10:33:44,228 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33728.80 MB 2025-02-14 10:33:44,228 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.90 MB 2025-02-14 10:33:44,228 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39319.50 MB 2025-02-14 10:33:44,228 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39319.50 MB 2025-02-14 10:33:44,228 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:33:44,228 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33946.74 MB 2025-02-14 10:33:44,229 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 10:33:44,229 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 10:33:44,229 - resource_logging.py:150 - __exit__ - DEBUG - Time: 33.23 seconds 2025-02-14 10:33:44,229 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:33:44,229 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19902.03 MB 2025-02-14 10:33:44,229 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33929.87 MB 2025-02-14 10:33:44,229 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14027.85 MB 2025-02-14 10:33:44,229 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47737.47 MB 2025-02-14 10:33:44,229 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39319.50 MB 2025-02-14 10:33:44,229 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8417.97 MB 2025-02-14 10:33:44,229 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33946.74 MB 2025-02-14 10:33:44,501 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 10:33:44,501 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 10:33:44,501 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 10:33:44,501 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:33:44,501 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33929.87 MB 2025-02-14 10:33:44,501 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24906.42 MB 2025-02-14 10:33:44,501 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9023.46 MB 2025-02-14 10:33:44,501 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39319.50 MB 2025-02-14 10:33:44,501 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39319.50 MB 2025-02-14 10:33:44,501 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:33:44,501 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36441.54 MB 2025-02-14 10:33:44,519 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 10:33:44,519 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 10:33:44,525 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 10:33:44,525 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 10:33:44,525 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:33:44,525 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:33:44,525 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24906.42 MB 2025-02-14 10:33:44,525 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33345.44 MB 2025-02-14 10:33:44,525 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 10:33:44,525 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39319.50 MB 2025-02-14 10:33:44,525 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47710.21 MB 2025-02-14 10:33:44,525 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 10:33:44,525 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33345.44 MB 2025-02-14 10:33:44,695 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 10:33:44,696 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:33:44,696 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 10:33:44,697 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:33:44,697 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 10:33:44,702 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 10:33:44,703 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:33:44,703 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 10:33:44,703 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 10:34:40,318 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:34:40,318 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 10:34:40,324 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 10:34:40,328 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:34:40,328 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 161, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 10:34:40,329 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:34:40,329 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 161, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 10:34:42,834 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 10:34:42,834 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 10:34:42,834 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.50 seconds 2025-02-14 10:34:42,834 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:34:42,834 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14090.58 MB 2025-02-14 10:34:42,834 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14660.35 MB 2025-02-14 10:34:42,834 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 569.77 MB 2025-02-14 10:34:42,834 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60295.22 MB 2025-02-14 10:34:42,834 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18324.91 MB 2025-02-14 10:34:42,834 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -41970.30 MB 2025-02-14 10:34:42,834 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23562.76 MB 2025-02-14 10:34:42,847 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 10:34:42,847 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 10:34:42,847 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:34:42,847 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:34:42,847 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14660.35 MB 2025-02-14 10:34:42,847 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14937.06 MB 2025-02-14 10:34:42,847 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 276.71 MB 2025-02-14 10:34:42,847 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18324.91 MB 2025-02-14 10:34:42,847 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18895.34 MB 2025-02-14 10:34:42,847 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 570.43 MB 2025-02-14 10:34:42,847 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16926.10 MB 2025-02-14 10:34:43,611 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 10:34:43,611 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 10:34:43,611 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.76 seconds 2025-02-14 10:34:43,611 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:34:43,611 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14937.06 MB 2025-02-14 10:34:43,611 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15150.72 MB 2025-02-14 10:34:43,611 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 213.66 MB 2025-02-14 10:34:43,611 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18895.34 MB 2025-02-14 10:34:43,611 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18895.34 MB 2025-02-14 10:34:43,611 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:34:43,611 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19108.53 MB 2025-02-14 10:34:43,619 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 10:34:43,619 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 10:34:43,619 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 10:34:43,619 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:34:43,619 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15150.66 MB 2025-02-14 10:34:43,619 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15911.01 MB 2025-02-14 10:34:43,619 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 760.35 MB 2025-02-14 10:34:43,619 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18895.34 MB 2025-02-14 10:34:43,619 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18895.34 MB 2025-02-14 10:34:43,619 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:34:43,619 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16481.53 MB 2025-02-14 10:34:43,705 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 10:34:43,705 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 10:34:43,705 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 10:34:43,705 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:34:43,705 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15911.01 MB 2025-02-14 10:34:43,705 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16813.39 MB 2025-02-14 10:34:43,705 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 902.39 MB 2025-02-14 10:34:43,705 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18895.34 MB 2025-02-14 10:34:43,705 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20422.07 MB 2025-02-14 10:34:43,705 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1526.73 MB 2025-02-14 10:34:43,705 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19044.93 MB 2025-02-14 10:34:43,706 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 10:34:43,706 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 10:34:43,706 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 10:34:43,706 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:34:43,706 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15150.66 MB 2025-02-14 10:34:43,706 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16813.39 MB 2025-02-14 10:34:43,706 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1662.74 MB 2025-02-14 10:34:43,706 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18895.34 MB 2025-02-14 10:34:43,706 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20422.07 MB 2025-02-14 10:34:43,706 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1526.73 MB 2025-02-14 10:34:43,706 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19044.93 MB 2025-02-14 10:34:43,773 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 10:34:43,773 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 10:34:43,773 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 10:34:43,773 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:34:43,773 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17430.65 MB 2025-02-14 10:34:43,773 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17739.36 MB 2025-02-14 10:34:43,773 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 308.72 MB 2025-02-14 10:34:43,773 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20422.07 MB 2025-02-14 10:34:43,773 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20585.64 MB 2025-02-14 10:34:43,773 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 163.58 MB 2025-02-14 10:34:43,773 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18032.94 MB 2025-02-14 10:34:43,783 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 10:34:43,783 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 10:34:43,783 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:34:43,783 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:34:43,783 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17905.56 MB 2025-02-14 10:34:43,783 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18134.23 MB 2025-02-14 10:34:43,783 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.67 MB 2025-02-14 10:34:43,783 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20585.64 MB 2025-02-14 10:34:43,783 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20585.64 MB 2025-02-14 10:34:43,783 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:34:43,783 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18154.33 MB 2025-02-14 10:34:43,784 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 10:34:43,784 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 10:34:43,784 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.45 seconds 2025-02-14 10:34:43,784 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:34:43,784 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13529.64 MB 2025-02-14 10:34:43,784 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18335.30 MB 2025-02-14 10:34:43,784 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4805.66 MB 2025-02-14 10:34:43,784 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60295.22 MB 2025-02-14 10:34:43,784 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20585.64 MB 2025-02-14 10:34:43,784 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -39709.57 MB 2025-02-14 10:34:43,784 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18335.30 MB 2025-02-14 10:34:44,049 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 10:34:44,049 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 10:34:44,049 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 10:34:44,049 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:34:44,049 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18335.30 MB 2025-02-14 10:34:44,049 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17406.12 MB 2025-02-14 10:34:44,049 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -929.18 MB 2025-02-14 10:34:44,049 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20585.64 MB 2025-02-14 10:34:44,049 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20585.64 MB 2025-02-14 10:34:44,049 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:34:44,049 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19139.04 MB 2025-02-14 10:34:44,067 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 10:34:44,067 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 10:34:44,074 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 10:34:44,074 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 10:34:44,074 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:34:44,074 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:34:44,074 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17406.12 MB 2025-02-14 10:34:44,074 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25845.14 MB 2025-02-14 10:34:44,074 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 10:34:44,074 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20585.64 MB 2025-02-14 10:34:44,074 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31075.60 MB 2025-02-14 10:34:44,074 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 10:34:44,074 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25845.14 MB 2025-02-14 10:34:44,227 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 10:34:44,228 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:34:44,229 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 10:34:44,229 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:34:44,229 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 10:34:44,234 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 10:34:44,235 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:34:44,235 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 10:34:44,235 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 10:35:42,427 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:35:42,427 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 10:35:42,432 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 10:35:42,435 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:35:42,436 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1307, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 10:35:42,436 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:35:42,437 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1307, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 10:36:02,393 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 10:36:02,393 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 10:36:02,393 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.95 seconds 2025-02-14 10:36:02,393 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:36:02,393 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22076.09 MB 2025-02-14 10:36:02,393 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26702.41 MB 2025-02-14 10:36:02,393 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4626.32 MB 2025-02-14 10:36:02,393 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43660.61 MB 2025-02-14 10:36:02,393 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38314.97 MB 2025-02-14 10:36:02,393 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5345.64 MB 2025-02-14 10:36:02,393 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35624.33 MB 2025-02-14 10:36:02,469 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 10:36:02,469 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 10:36:02,469 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 10:36:02,469 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:36:02,469 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26702.41 MB 2025-02-14 10:36:02,469 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22572.54 MB 2025-02-14 10:36:02,469 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4129.87 MB 2025-02-14 10:36:02,469 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38314.97 MB 2025-02-14 10:36:02,469 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47496.30 MB 2025-02-14 10:36:02,469 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9181.33 MB 2025-02-14 10:36:02,469 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40548.11 MB 2025-02-14 10:36:04,377 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 10:36:04,377 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 10:36:04,377 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 10:36:04,377 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:36:04,377 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22572.54 MB 2025-02-14 10:36:04,377 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23103.38 MB 2025-02-14 10:36:04,377 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 10:36:04,377 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47496.30 MB 2025-02-14 10:36:04,377 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33688.65 MB 2025-02-14 10:36:04,377 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13807.65 MB 2025-02-14 10:36:04,377 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27082.71 MB 2025-02-14 10:36:04,390 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 10:36:04,390 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 10:36:04,390 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:36:04,390 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:36:04,390 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23103.38 MB 2025-02-14 10:36:04,390 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24992.91 MB 2025-02-14 10:36:04,390 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 10:36:04,390 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33688.65 MB 2025-02-14 10:36:04,390 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33688.65 MB 2025-02-14 10:36:04,390 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:36:04,390 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26410.34 MB 2025-02-14 10:36:04,605 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 10:36:04,605 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 10:36:04,605 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 10:36:04,605 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:36:04,605 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24992.91 MB 2025-02-14 10:36:04,605 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27234.77 MB 2025-02-14 10:36:04,605 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 10:36:04,605 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33688.65 MB 2025-02-14 10:36:04,605 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36047.95 MB 2025-02-14 10:36:04,605 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2359.30 MB 2025-02-14 10:36:04,605 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32779.05 MB 2025-02-14 10:36:04,606 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 10:36:04,606 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 10:36:04,606 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 10:36:04,606 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:36:04,606 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23103.38 MB 2025-02-14 10:36:04,606 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27234.77 MB 2025-02-14 10:36:04,606 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 10:36:04,606 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33688.65 MB 2025-02-14 10:36:04,606 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36047.95 MB 2025-02-14 10:36:04,606 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2359.30 MB 2025-02-14 10:36:04,606 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32779.05 MB 2025-02-14 10:36:04,782 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 10:36:04,782 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 10:36:04,782 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 10:36:04,782 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:36:04,782 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28768.31 MB 2025-02-14 10:36:04,782 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29535.31 MB 2025-02-14 10:36:04,782 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 10:36:04,782 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36047.95 MB 2025-02-14 10:36:04,782 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36463.18 MB 2025-02-14 10:36:04,782 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 10:36:04,782 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30243.10 MB 2025-02-14 10:36:04,802 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 10:36:04,802 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 10:36:04,802 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:36:04,802 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:36:04,802 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29948.20 MB 2025-02-14 10:36:04,802 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30175.74 MB 2025-02-14 10:36:04,802 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.54 MB 2025-02-14 10:36:04,802 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36463.18 MB 2025-02-14 10:36:04,802 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36463.18 MB 2025-02-14 10:36:04,802 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:36:04,802 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30419.63 MB 2025-02-14 10:36:04,803 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 10:36:04,803 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 10:36:04,803 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.36 seconds 2025-02-14 10:36:04,803 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:36:04,803 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17522.40 MB 2025-02-14 10:36:04,803 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30376.12 MB 2025-02-14 10:36:04,803 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12853.72 MB 2025-02-14 10:36:04,803 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43660.61 MB 2025-02-14 10:36:04,803 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36463.18 MB 2025-02-14 10:36:04,803 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7197.43 MB 2025-02-14 10:36:04,803 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30419.63 MB 2025-02-14 10:36:05,071 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 10:36:05,071 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 10:36:05,071 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 10:36:05,071 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:36:05,071 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30376.12 MB 2025-02-14 10:36:05,071 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22516.12 MB 2025-02-14 10:36:05,071 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7860.00 MB 2025-02-14 10:36:05,071 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36463.18 MB 2025-02-14 10:36:05,071 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36463.18 MB 2025-02-14 10:36:05,071 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:36:05,071 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32879.19 MB 2025-02-14 10:36:05,088 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8134, cut from 8136 2025-02-14 10:36:05,089 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 10:36:05,095 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 10:36:05,095 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 10:36:05,095 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:36:05,095 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:36:05,095 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22516.12 MB 2025-02-14 10:36:05,095 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30925.93 MB 2025-02-14 10:36:05,095 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8409.81 MB 2025-02-14 10:36:05,095 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36463.18 MB 2025-02-14 10:36:05,095 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40642.81 MB 2025-02-14 10:36:05,095 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4179.62 MB 2025-02-14 10:36:05,095 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30925.93 MB 2025-02-14 10:36:05,266 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7926] 2025-02-14 10:36:05,267 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:36:05,267 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 10:36:05,270 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:36:05,270 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 10:36:05,275 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 10:36:05,276 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:36:05,276 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 10:36:05,276 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 10:36:42,429 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:36:42,429 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 10:36:42,434 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 10:36:42,438 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:36:42,438 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1410, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 10:36:42,439 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:36:42,439 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1410, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 10:37:04,112 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 10:37:04,112 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 10:37:04,112 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.67 seconds 2025-02-14 10:37:04,112 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:37:04,112 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22793.81 MB 2025-02-14 10:37:04,112 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27783.72 MB 2025-02-14 10:37:04,113 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4989.91 MB 2025-02-14 10:37:04,113 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53183.77 MB 2025-02-14 10:37:04,113 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38621.15 MB 2025-02-14 10:37:04,113 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14562.62 MB 2025-02-14 10:37:04,113 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36795.03 MB 2025-02-14 10:37:04,193 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 10:37:04,193 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 10:37:04,193 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 10:37:04,193 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:37:04,193 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27783.72 MB 2025-02-14 10:37:04,193 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23108.00 MB 2025-02-14 10:37:04,193 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4675.72 MB 2025-02-14 10:37:04,193 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38621.15 MB 2025-02-14 10:37:04,193 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47718.60 MB 2025-02-14 10:37:04,193 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9097.45 MB 2025-02-14 10:37:04,193 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41469.78 MB 2025-02-14 10:37:06,116 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 10:37:06,116 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 10:37:06,116 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 10:37:06,116 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:37:06,116 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23108.00 MB 2025-02-14 10:37:06,116 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23638.84 MB 2025-02-14 10:37:06,116 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 10:37:06,116 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47718.60 MB 2025-02-14 10:37:06,116 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33629.93 MB 2025-02-14 10:37:06,116 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14088.67 MB 2025-02-14 10:37:06,116 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27618.18 MB 2025-02-14 10:37:06,130 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 10:37:06,130 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 10:37:06,130 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:37:06,130 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:37:06,130 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23638.84 MB 2025-02-14 10:37:06,130 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25528.38 MB 2025-02-14 10:37:06,130 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 10:37:06,130 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33629.93 MB 2025-02-14 10:37:06,130 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33629.93 MB 2025-02-14 10:37:06,130 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:37:06,130 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26945.81 MB 2025-02-14 10:37:06,348 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 10:37:06,348 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 10:37:06,348 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 10:37:06,348 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:37:06,348 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25528.38 MB 2025-02-14 10:37:06,348 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27770.23 MB 2025-02-14 10:37:06,348 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 10:37:06,348 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33629.93 MB 2025-02-14 10:37:06,348 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37404.80 MB 2025-02-14 10:37:06,348 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 10:37:06,348 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33314.51 MB 2025-02-14 10:37:06,349 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 10:37:06,349 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 10:37:06,349 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 10:37:06,349 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:37:06,349 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23638.84 MB 2025-02-14 10:37:06,349 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27770.23 MB 2025-02-14 10:37:06,349 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 10:37:06,349 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33629.93 MB 2025-02-14 10:37:06,349 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37404.80 MB 2025-02-14 10:37:06,349 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 10:37:06,349 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33314.51 MB 2025-02-14 10:37:06,522 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 10:37:06,522 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 10:37:06,523 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 10:37:06,523 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:37:06,523 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29303.77 MB 2025-02-14 10:37:06,523 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30070.78 MB 2025-02-14 10:37:06,523 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 10:37:06,523 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37404.80 MB 2025-02-14 10:37:06,523 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37820.04 MB 2025-02-14 10:37:06,523 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 10:37:06,523 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30778.57 MB 2025-02-14 10:37:06,542 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 10:37:06,542 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 10:37:06,542 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:37:06,542 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:37:06,542 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30483.67 MB 2025-02-14 10:37:06,542 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30712.45 MB 2025-02-14 10:37:06,542 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.79 MB 2025-02-14 10:37:06,542 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37820.04 MB 2025-02-14 10:37:06,542 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37820.04 MB 2025-02-14 10:37:06,542 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:37:06,542 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30953.44 MB 2025-02-14 10:37:06,543 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 10:37:06,543 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 10:37:06,543 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.10 seconds 2025-02-14 10:37:06,543 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:37:06,543 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17881.26 MB 2025-02-14 10:37:06,543 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30913.16 MB 2025-02-14 10:37:06,543 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13031.90 MB 2025-02-14 10:37:06,543 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53183.77 MB 2025-02-14 10:37:06,543 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37820.04 MB 2025-02-14 10:37:06,543 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15363.74 MB 2025-02-14 10:37:06,543 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30953.44 MB 2025-02-14 10:37:06,811 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 10:37:06,811 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 10:37:06,811 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 10:37:06,811 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:37:06,811 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30913.16 MB 2025-02-14 10:37:06,811 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22879.93 MB 2025-02-14 10:37:06,811 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8033.22 MB 2025-02-14 10:37:06,811 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37820.04 MB 2025-02-14 10:37:06,811 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37820.04 MB 2025-02-14 10:37:06,812 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:37:06,812 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33420.22 MB 2025-02-14 10:37:06,830 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8147, cut from 8149 2025-02-14 10:37:06,830 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 10:37:06,843 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 10:37:06,843 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 10:37:06,843 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 10:37:06,843 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:37:06,843 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22879.93 MB 2025-02-14 10:37:06,843 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31303.14 MB 2025-02-14 10:37:06,843 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8423.21 MB 2025-02-14 10:37:06,843 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37820.04 MB 2025-02-14 10:37:06,844 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46196.06 MB 2025-02-14 10:37:06,844 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8376.03 MB 2025-02-14 10:37:06,844 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31303.14 MB 2025-02-14 10:37:07,012 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7939] 2025-02-14 10:37:07,013 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:37:07,013 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 10:37:07,014 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:37:07,014 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 10:37:07,019 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 10:37:07,020 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:37:07,020 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 10:37:07,020 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 10:37:15,629 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:37:15,629 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 10:37:15,634 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 10:37:15,637 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:37:15,637 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 972, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 10:37:15,638 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:37:15,638 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 972, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 10:37:30,808 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 10:37:30,808 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 10:37:30,808 - resource_logging.py:150 - __exit__ - DEBUG - Time: 15.16 seconds 2025-02-14 10:37:30,808 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:37:30,808 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19741.76 MB 2025-02-14 10:37:30,808 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23181.61 MB 2025-02-14 10:37:30,808 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3439.85 MB 2025-02-14 10:37:30,808 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54572.09 MB 2025-02-14 10:37:30,808 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28710.01 MB 2025-02-14 10:37:30,808 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25862.08 MB 2025-02-14 10:37:30,808 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32158.34 MB 2025-02-14 10:37:30,860 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 10:37:30,860 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 10:37:30,860 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-14 10:37:30,860 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:37:30,861 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23181.61 MB 2025-02-14 10:37:30,861 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20830.98 MB 2025-02-14 10:37:30,861 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2350.64 MB 2025-02-14 10:37:30,861 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28710.01 MB 2025-02-14 10:37:30,861 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36228.30 MB 2025-02-14 10:37:30,861 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7518.29 MB 2025-02-14 10:37:30,861 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32134.60 MB 2025-02-14 10:37:32,791 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 10:37:32,791 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 10:37:32,791 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 10:37:32,791 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:37:32,791 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20830.98 MB 2025-02-14 10:37:32,791 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21361.82 MB 2025-02-14 10:37:32,791 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 10:37:32,791 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36228.30 MB 2025-02-14 10:37:32,791 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26684.16 MB 2025-02-14 10:37:32,791 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9544.14 MB 2025-02-14 10:37:32,791 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25341.15 MB 2025-02-14 10:37:32,805 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 10:37:32,805 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 10:37:32,805 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:37:32,805 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:37:32,805 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21361.82 MB 2025-02-14 10:37:32,805 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23251.35 MB 2025-02-14 10:37:32,805 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 10:37:32,805 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26684.16 MB 2025-02-14 10:37:32,805 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27627.88 MB 2025-02-14 10:37:32,805 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 10:37:32,805 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24668.78 MB 2025-02-14 10:37:33,017 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 10:37:33,017 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 10:37:33,017 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 10:37:33,017 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:37:33,017 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23251.35 MB 2025-02-14 10:37:33,017 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25493.21 MB 2025-02-14 10:37:33,017 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 10:37:33,017 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27627.88 MB 2025-02-14 10:37:33,017 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33762.05 MB 2025-02-14 10:37:33,017 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 10:37:33,017 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31037.49 MB 2025-02-14 10:37:33,018 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 10:37:33,018 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 10:37:33,018 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 10:37:33,018 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:37:33,018 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21361.82 MB 2025-02-14 10:37:33,018 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25493.21 MB 2025-02-14 10:37:33,018 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 10:37:33,018 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26684.16 MB 2025-02-14 10:37:33,018 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33762.05 MB 2025-02-14 10:37:33,018 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7077.89 MB 2025-02-14 10:37:33,018 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31037.49 MB 2025-02-14 10:37:33,184 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 10:37:33,184 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 10:37:33,184 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 10:37:33,184 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:37:33,184 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27026.75 MB 2025-02-14 10:37:33,184 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27793.75 MB 2025-02-14 10:37:33,184 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 10:37:33,184 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33762.05 MB 2025-02-14 10:37:33,184 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34179.38 MB 2025-02-14 10:37:33,184 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 10:37:33,184 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28501.54 MB 2025-02-14 10:37:33,203 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 10:37:33,203 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 10:37:33,203 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:37:33,203 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:37:33,203 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28206.64 MB 2025-02-14 10:37:33,203 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28436.72 MB 2025-02-14 10:37:33,203 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 230.08 MB 2025-02-14 10:37:33,203 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34179.38 MB 2025-02-14 10:37:33,203 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34179.38 MB 2025-02-14 10:37:33,203 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:37:33,203 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28640.78 MB 2025-02-14 10:37:33,204 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 10:37:33,204 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 10:37:33,204 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.56 seconds 2025-02-14 10:37:33,204 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:37:33,204 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16355.23 MB 2025-02-14 10:37:33,204 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28637.79 MB 2025-02-14 10:37:33,204 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12282.56 MB 2025-02-14 10:37:33,204 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54572.09 MB 2025-02-14 10:37:33,204 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34179.38 MB 2025-02-14 10:37:33,204 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20392.71 MB 2025-02-14 10:37:33,204 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28640.78 MB 2025-02-14 10:37:33,474 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 10:37:33,474 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 10:37:33,474 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 10:37:33,474 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:37:33,474 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28637.79 MB 2025-02-14 10:37:33,474 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21359.62 MB 2025-02-14 10:37:33,474 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7278.17 MB 2025-02-14 10:37:33,474 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34179.38 MB 2025-02-14 10:37:33,474 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34179.38 MB 2025-02-14 10:37:33,474 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:37:33,474 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31149.46 MB 2025-02-14 10:37:33,492 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 10:37:33,492 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 10:37:33,498 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 10:37:33,498 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 10:37:33,498 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:37:33,499 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:37:33,499 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21359.62 MB 2025-02-14 10:37:33,499 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29798.64 MB 2025-02-14 10:37:33,499 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 10:37:33,499 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34179.38 MB 2025-02-14 10:37:33,499 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44669.34 MB 2025-02-14 10:37:33,499 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 10:37:33,499 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29798.64 MB 2025-02-14 10:37:33,661 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 10:37:33,662 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:37:33,662 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 10:37:33,663 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:37:33,663 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 10:37:33,668 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 10:37:33,669 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:37:33,669 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 10:37:33,669 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 10:38:14,009 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:38:14,009 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 10:38:14,015 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 10:38:14,019 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:38:14,019 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 161, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 10:38:14,020 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:38:14,020 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 161, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 10:38:16,556 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 10:38:16,557 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 10:38:16,557 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.53 seconds 2025-02-14 10:38:16,557 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:38:16,557 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14090.58 MB 2025-02-14 10:38:16,557 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14660.35 MB 2025-02-14 10:38:16,557 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 569.77 MB 2025-02-14 10:38:16,557 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57254.35 MB 2025-02-14 10:38:16,557 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17754.49 MB 2025-02-14 10:38:16,557 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -39499.86 MB 2025-02-14 10:38:16,557 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23562.77 MB 2025-02-14 10:38:16,569 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 10:38:16,569 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 10:38:16,569 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:38:16,569 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:38:16,569 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14660.35 MB 2025-02-14 10:38:16,569 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14937.06 MB 2025-02-14 10:38:16,569 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 276.71 MB 2025-02-14 10:38:16,569 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17754.49 MB 2025-02-14 10:38:16,569 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18324.91 MB 2025-02-14 10:38:16,569 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 570.43 MB 2025-02-14 10:38:16,569 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16926.98 MB 2025-02-14 10:38:17,344 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 10:38:17,344 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 10:38:17,344 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.77 seconds 2025-02-14 10:38:17,344 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:38:17,344 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14937.06 MB 2025-02-14 10:38:17,344 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15150.72 MB 2025-02-14 10:38:17,344 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 213.66 MB 2025-02-14 10:38:17,344 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18324.91 MB 2025-02-14 10:38:17,344 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17479.76 MB 2025-02-14 10:38:17,344 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -845.15 MB 2025-02-14 10:38:17,344 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19108.53 MB 2025-02-14 10:38:17,352 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 10:38:17,352 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 10:38:17,352 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:38:17,352 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:38:17,352 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15150.66 MB 2025-02-14 10:38:17,352 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15911.01 MB 2025-02-14 10:38:17,352 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 760.35 MB 2025-02-14 10:38:17,352 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17479.76 MB 2025-02-14 10:38:17,352 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17861.44 MB 2025-02-14 10:38:17,352 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 381.68 MB 2025-02-14 10:38:17,352 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16481.53 MB 2025-02-14 10:38:17,438 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 10:38:17,438 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 10:38:17,438 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 10:38:17,438 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:38:17,438 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15911.01 MB 2025-02-14 10:38:17,438 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16813.39 MB 2025-02-14 10:38:17,438 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 902.39 MB 2025-02-14 10:38:17,438 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17861.44 MB 2025-02-14 10:38:17,438 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20342.37 MB 2025-02-14 10:38:17,438 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2480.93 MB 2025-02-14 10:38:17,438 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19047.68 MB 2025-02-14 10:38:17,439 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 10:38:17,439 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 10:38:17,439 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 10:38:17,439 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:38:17,439 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15150.66 MB 2025-02-14 10:38:17,439 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16813.39 MB 2025-02-14 10:38:17,439 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1662.74 MB 2025-02-14 10:38:17,439 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17479.76 MB 2025-02-14 10:38:17,439 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20342.37 MB 2025-02-14 10:38:17,439 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2862.61 MB 2025-02-14 10:38:17,439 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19047.68 MB 2025-02-14 10:38:17,505 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 10:38:17,505 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 10:38:17,505 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 10:38:17,505 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:38:17,505 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17430.65 MB 2025-02-14 10:38:17,506 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17741.20 MB 2025-02-14 10:38:17,506 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 310.55 MB 2025-02-14 10:38:17,506 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20342.37 MB 2025-02-14 10:38:17,506 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20505.95 MB 2025-02-14 10:38:17,506 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 163.58 MB 2025-02-14 10:38:17,506 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18034.76 MB 2025-02-14 10:38:17,515 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 10:38:17,515 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 10:38:17,515 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:38:17,515 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:38:17,515 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17907.39 MB 2025-02-14 10:38:17,515 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18133.99 MB 2025-02-14 10:38:17,515 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.59 MB 2025-02-14 10:38:17,515 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20505.95 MB 2025-02-14 10:38:17,515 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20505.95 MB 2025-02-14 10:38:17,515 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:38:17,515 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18144.74 MB 2025-02-14 10:38:17,516 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 10:38:17,516 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 10:38:17,516 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.49 seconds 2025-02-14 10:38:17,516 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:38:17,516 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13529.64 MB 2025-02-14 10:38:17,516 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18334.64 MB 2025-02-14 10:38:17,516 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4805.00 MB 2025-02-14 10:38:17,516 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57254.35 MB 2025-02-14 10:38:17,516 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20505.95 MB 2025-02-14 10:38:17,516 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -36748.39 MB 2025-02-14 10:38:17,516 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18334.64 MB 2025-02-14 10:38:17,784 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 10:38:17,784 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 10:38:17,784 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 10:38:17,784 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:38:17,784 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18334.64 MB 2025-02-14 10:38:17,784 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17401.22 MB 2025-02-14 10:38:17,784 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -933.42 MB 2025-02-14 10:38:17,784 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20505.95 MB 2025-02-14 10:38:17,784 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20505.95 MB 2025-02-14 10:38:17,784 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:38:17,784 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19136.70 MB 2025-02-14 10:38:17,802 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8145, cut from 8147 2025-02-14 10:38:17,802 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 10:38:17,808 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 10:38:17,808 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 10:38:17,808 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:38:17,808 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:38:17,808 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17401.22 MB 2025-02-14 10:38:17,808 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25823.18 MB 2025-02-14 10:38:17,808 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8421.96 MB 2025-02-14 10:38:17,808 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20505.95 MB 2025-02-14 10:38:17,808 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30970.74 MB 2025-02-14 10:38:17,808 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10464.79 MB 2025-02-14 10:38:17,808 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25823.18 MB 2025-02-14 10:38:17,959 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7937] 2025-02-14 10:38:17,960 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:38:17,960 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 10:38:17,961 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:38:17,961 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 10:38:17,966 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 10:38:17,967 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:38:17,967 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 10:38:17,967 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 10:38:59,498 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:38:59,498 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 10:38:59,503 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 10:38:59,506 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:38:59,506 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 898, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 10:38:59,507 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:38:59,507 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 898, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 10:39:13,366 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 10:39:13,366 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 10:39:13,366 - resource_logging.py:150 - __exit__ - DEBUG - Time: 13.85 seconds 2025-02-14 10:39:13,366 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:39:13,366 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19226.11 MB 2025-02-14 10:39:13,366 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22404.09 MB 2025-02-14 10:39:13,366 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3177.97 MB 2025-02-14 10:39:13,366 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39342.57 MB 2025-02-14 10:39:13,366 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30555.50 MB 2025-02-14 10:39:13,366 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8787.07 MB 2025-02-14 10:39:13,366 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31415.39 MB 2025-02-14 10:39:13,418 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 10:39:13,418 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 10:39:13,418 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-14 10:39:13,418 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:39:13,418 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22404.09 MB 2025-02-14 10:39:13,418 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20446.27 MB 2025-02-14 10:39:13,418 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1957.81 MB 2025-02-14 10:39:13,418 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30555.50 MB 2025-02-14 10:39:13,418 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37658.56 MB 2025-02-14 10:39:13,418 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7103.05 MB 2025-02-14 10:39:13,418 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32346.43 MB 2025-02-14 10:39:15,323 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 10:39:15,324 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 10:39:15,324 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.90 seconds 2025-02-14 10:39:15,324 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:39:15,324 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20446.27 MB 2025-02-14 10:39:15,324 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20977.11 MB 2025-02-14 10:39:15,324 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 10:39:15,324 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37658.56 MB 2025-02-14 10:39:15,324 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28791.80 MB 2025-02-14 10:39:15,324 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8866.76 MB 2025-02-14 10:39:15,324 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24956.45 MB 2025-02-14 10:39:15,337 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 10:39:15,337 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 10:39:15,337 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:39:15,337 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:39:15,337 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20977.11 MB 2025-02-14 10:39:15,337 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22866.65 MB 2025-02-14 10:39:15,337 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 10:39:15,337 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28791.80 MB 2025-02-14 10:39:15,337 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28791.80 MB 2025-02-14 10:39:15,337 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:39:15,337 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24284.08 MB 2025-02-14 10:39:15,546 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 10:39:15,546 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 10:39:15,546 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 10:39:15,546 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:39:15,546 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22866.65 MB 2025-02-14 10:39:15,546 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25108.50 MB 2025-02-14 10:39:15,546 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 10:39:15,546 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28791.80 MB 2025-02-14 10:39:15,546 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33510.39 MB 2025-02-14 10:39:15,546 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 10:39:15,546 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30652.79 MB 2025-02-14 10:39:15,546 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 10:39:15,546 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 10:39:15,546 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 10:39:15,546 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:39:15,546 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20977.11 MB 2025-02-14 10:39:15,546 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25108.50 MB 2025-02-14 10:39:15,546 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 10:39:15,547 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28791.80 MB 2025-02-14 10:39:15,547 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33510.39 MB 2025-02-14 10:39:15,547 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 10:39:15,547 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30652.79 MB 2025-02-14 10:39:15,716 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 10:39:15,716 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 10:39:15,716 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 10:39:15,716 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:39:15,716 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26642.05 MB 2025-02-14 10:39:15,716 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27409.05 MB 2025-02-14 10:39:15,716 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 10:39:15,716 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33510.39 MB 2025-02-14 10:39:15,716 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33923.53 MB 2025-02-14 10:39:15,716 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 10:39:15,716 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28116.84 MB 2025-02-14 10:39:15,735 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 10:39:15,735 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 10:39:15,735 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:39:15,735 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:39:15,735 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27821.94 MB 2025-02-14 10:39:15,735 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28049.96 MB 2025-02-14 10:39:15,735 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.03 MB 2025-02-14 10:39:15,735 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33923.53 MB 2025-02-14 10:39:15,735 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33923.53 MB 2025-02-14 10:39:15,735 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:39:15,735 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28251.10 MB 2025-02-14 10:39:15,736 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 10:39:15,736 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 10:39:15,736 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.23 seconds 2025-02-14 10:39:15,736 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:39:15,736 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16097.41 MB 2025-02-14 10:39:15,736 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28249.91 MB 2025-02-14 10:39:15,736 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12152.50 MB 2025-02-14 10:39:15,736 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39342.57 MB 2025-02-14 10:39:15,736 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33923.53 MB 2025-02-14 10:39:15,736 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5419.04 MB 2025-02-14 10:39:15,736 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28251.10 MB 2025-02-14 10:39:16,006 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 10:39:16,006 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 10:39:16,006 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 10:39:16,006 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:39:16,006 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28249.91 MB 2025-02-14 10:39:16,006 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21084.28 MB 2025-02-14 10:39:16,006 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7165.63 MB 2025-02-14 10:39:16,006 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33923.53 MB 2025-02-14 10:39:16,006 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33923.53 MB 2025-02-14 10:39:16,006 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:39:16,006 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30747.44 MB 2025-02-14 10:39:16,023 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8116, cut from 8118 2025-02-14 10:39:16,024 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 10:39:16,030 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 10:39:16,030 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 10:39:16,030 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:39:16,030 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:39:16,030 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21084.28 MB 2025-02-14 10:39:16,030 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29476.70 MB 2025-02-14 10:39:16,030 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8392.42 MB 2025-02-14 10:39:16,030 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33923.53 MB 2025-02-14 10:39:16,030 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42266.00 MB 2025-02-14 10:39:16,030 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8342.47 MB 2025-02-14 10:39:16,030 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29476.70 MB 2025-02-14 10:39:16,191 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7908] 2025-02-14 10:39:16,193 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:39:16,193 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 10:39:16,194 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:39:16,194 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 10:39:16,198 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 10:39:16,199 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:39:16,199 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 10:39:16,199 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 10:39:44,302 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:39:44,303 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 10:39:44,307 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 10:39:44,311 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:39:44,311 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 986, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 10:39:44,312 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:39:44,312 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 986, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 10:39:59,694 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 10:39:59,694 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 10:39:59,694 - resource_logging.py:150 - __exit__ - DEBUG - Time: 15.38 seconds 2025-02-14 10:39:59,694 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:39:59,694 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19839.31 MB 2025-02-14 10:39:59,694 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23328.97 MB 2025-02-14 10:39:59,694 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3489.66 MB 2025-02-14 10:39:59,694 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50608.47 MB 2025-02-14 10:39:59,694 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30863.79 MB 2025-02-14 10:39:59,694 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19744.69 MB 2025-02-14 10:39:59,694 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32255.89 MB 2025-02-14 10:39:59,754 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 10:39:59,754 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 10:39:59,754 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 10:39:59,754 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:39:59,754 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23328.97 MB 2025-02-14 10:39:59,754 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20903.76 MB 2025-02-14 10:39:59,754 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2425.22 MB 2025-02-14 10:39:59,754 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30863.79 MB 2025-02-14 10:39:59,754 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39067.84 MB 2025-02-14 10:39:59,754 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8204.06 MB 2025-02-14 10:39:59,754 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34373.83 MB 2025-02-14 10:40:01,676 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 10:40:01,676 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 10:40:01,676 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 10:40:01,676 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:40:01,676 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20903.76 MB 2025-02-14 10:40:01,676 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21434.60 MB 2025-02-14 10:40:01,676 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 10:40:01,676 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39067.84 MB 2025-02-14 10:40:01,676 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26696.74 MB 2025-02-14 10:40:01,676 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12371.10 MB 2025-02-14 10:40:01,676 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25413.93 MB 2025-02-14 10:40:01,691 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 10:40:01,692 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 10:40:01,692 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:40:01,692 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:40:01,692 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21434.60 MB 2025-02-14 10:40:01,692 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23324.13 MB 2025-02-14 10:40:01,692 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 10:40:01,692 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26696.74 MB 2025-02-14 10:40:01,692 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27640.46 MB 2025-02-14 10:40:01,692 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 10:40:01,692 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24741.56 MB 2025-02-14 10:40:01,903 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 10:40:01,904 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 10:40:01,904 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 10:40:01,904 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:40:01,904 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23324.13 MB 2025-02-14 10:40:01,904 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25565.99 MB 2025-02-14 10:40:01,904 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 10:40:01,904 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27640.46 MB 2025-02-14 10:40:01,904 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33302.77 MB 2025-02-14 10:40:01,904 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 10:40:01,904 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31110.27 MB 2025-02-14 10:40:01,904 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 10:40:01,904 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 10:40:01,904 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 10:40:01,904 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:40:01,904 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21434.60 MB 2025-02-14 10:40:01,904 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25565.99 MB 2025-02-14 10:40:01,904 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 10:40:01,904 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26696.74 MB 2025-02-14 10:40:01,904 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33302.77 MB 2025-02-14 10:40:01,904 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 10:40:01,904 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31110.27 MB 2025-02-14 10:40:02,073 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 10:40:02,073 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 10:40:02,073 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 10:40:02,073 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:40:02,073 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27099.53 MB 2025-02-14 10:40:02,073 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27866.53 MB 2025-02-14 10:40:02,073 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 10:40:02,073 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33302.77 MB 2025-02-14 10:40:02,073 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33718.01 MB 2025-02-14 10:40:02,073 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 10:40:02,074 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28574.32 MB 2025-02-14 10:40:02,093 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 10:40:02,093 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 10:40:02,093 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:40:02,093 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:40:02,093 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28279.42 MB 2025-02-14 10:40:02,093 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28507.99 MB 2025-02-14 10:40:02,093 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.57 MB 2025-02-14 10:40:02,093 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33718.01 MB 2025-02-14 10:40:02,093 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33718.01 MB 2025-02-14 10:40:02,093 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:40:02,093 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28742.57 MB 2025-02-14 10:40:02,094 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 10:40:02,094 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 10:40:02,094 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.78 seconds 2025-02-14 10:40:02,094 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:40:02,094 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16404.01 MB 2025-02-14 10:40:02,094 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28708.47 MB 2025-02-14 10:40:02,094 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12304.46 MB 2025-02-14 10:40:02,094 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50608.47 MB 2025-02-14 10:40:02,094 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33718.01 MB 2025-02-14 10:40:02,094 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16890.46 MB 2025-02-14 10:40:02,094 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28742.57 MB 2025-02-14 10:40:02,365 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 10:40:02,365 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 10:40:02,365 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 10:40:02,365 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:40:02,365 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28708.47 MB 2025-02-14 10:40:02,365 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21399.26 MB 2025-02-14 10:40:02,365 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7309.22 MB 2025-02-14 10:40:02,365 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33718.01 MB 2025-02-14 10:40:02,365 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33718.01 MB 2025-02-14 10:40:02,365 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:40:02,365 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31212.77 MB 2025-02-14 10:40:02,383 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8138, cut from 8140 2025-02-14 10:40:02,383 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 10:40:02,389 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 10:40:02,389 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 10:40:02,389 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:40:02,389 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:40:02,389 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21399.26 MB 2025-02-14 10:40:02,389 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29813.24 MB 2025-02-14 10:40:02,389 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8413.98 MB 2025-02-14 10:40:02,389 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33718.01 MB 2025-02-14 10:40:02,389 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42083.55 MB 2025-02-14 10:40:02,389 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8365.54 MB 2025-02-14 10:40:02,389 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29813.24 MB 2025-02-14 10:40:02,552 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7930] 2025-02-14 10:40:02,554 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:40:02,554 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 10:40:02,555 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:40:02,555 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 10:40:02,559 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 10:40:02,561 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:40:02,561 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 10:40:02,561 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 10:40:45,784 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:40:45,785 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 10:40:45,790 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 10:40:45,793 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:40:45,793 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 674, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 10:40:45,794 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:40:45,794 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 674, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 10:40:56,239 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 10:40:56,239 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 10:40:56,239 - resource_logging.py:150 - __exit__ - DEBUG - Time: 10.44 seconds 2025-02-14 10:40:56,239 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:40:56,239 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17665.25 MB 2025-02-14 10:40:56,239 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20050.49 MB 2025-02-14 10:40:56,239 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2385.25 MB 2025-02-14 10:40:56,239 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54630.81 MB 2025-02-14 10:40:56,239 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24891.10 MB 2025-02-14 10:40:56,240 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29739.71 MB 2025-02-14 10:40:56,240 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28948.56 MB 2025-02-14 10:40:56,284 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 10:40:56,284 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 10:40:56,284 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-14 10:40:56,284 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:40:56,284 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20050.49 MB 2025-02-14 10:40:56,284 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19282.81 MB 2025-02-14 10:40:56,284 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -767.68 MB 2025-02-14 10:40:56,284 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24891.10 MB 2025-02-14 10:40:56,284 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31377.59 MB 2025-02-14 10:40:56,284 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6486.49 MB 2025-02-14 10:40:56,284 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28645.48 MB 2025-02-14 10:40:58,208 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 10:40:58,208 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 10:40:58,208 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 10:40:58,208 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:40:58,208 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19282.81 MB 2025-02-14 10:40:58,208 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19813.66 MB 2025-02-14 10:40:58,208 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 10:40:58,208 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31377.59 MB 2025-02-14 10:40:58,208 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24628.95 MB 2025-02-14 10:40:58,208 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6748.64 MB 2025-02-14 10:40:58,208 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23792.99 MB 2025-02-14 10:40:58,222 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 10:40:58,222 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 10:40:58,222 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:40:58,222 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:40:58,222 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19813.66 MB 2025-02-14 10:40:58,222 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21703.19 MB 2025-02-14 10:40:58,222 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 10:40:58,222 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24628.95 MB 2025-02-14 10:40:58,222 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26516.39 MB 2025-02-14 10:40:58,222 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 10:40:58,222 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23120.62 MB 2025-02-14 10:40:58,434 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 10:40:58,434 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 10:40:58,434 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 10:40:58,434 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:40:58,434 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21703.19 MB 2025-02-14 10:40:58,434 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23945.70 MB 2025-02-14 10:40:58,434 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2242.51 MB 2025-02-14 10:40:58,434 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26516.39 MB 2025-02-14 10:40:58,434 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32178.70 MB 2025-02-14 10:40:58,434 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 10:40:58,434 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29489.98 MB 2025-02-14 10:40:58,435 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 10:40:58,435 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 10:40:58,435 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 10:40:58,435 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:40:58,435 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19813.66 MB 2025-02-14 10:40:58,435 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23945.70 MB 2025-02-14 10:40:58,435 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4132.05 MB 2025-02-14 10:40:58,435 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24628.95 MB 2025-02-14 10:40:58,435 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32178.70 MB 2025-02-14 10:40:58,435 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-14 10:40:58,435 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29489.98 MB 2025-02-14 10:40:58,604 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 10:40:58,604 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 10:40:58,604 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 10:40:58,604 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:40:58,604 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25479.24 MB 2025-02-14 10:40:58,604 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26246.25 MB 2025-02-14 10:40:58,604 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 10:40:58,604 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32178.70 MB 2025-02-14 10:40:58,604 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32596.03 MB 2025-02-14 10:40:58,604 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 10:40:58,604 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26954.03 MB 2025-02-14 10:40:58,624 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 10:40:58,624 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 10:40:58,624 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:40:58,624 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:40:58,624 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26659.13 MB 2025-02-14 10:40:58,624 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26888.35 MB 2025-02-14 10:40:58,624 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.22 MB 2025-02-14 10:40:58,624 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32596.03 MB 2025-02-14 10:40:58,624 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32596.03 MB 2025-02-14 10:40:58,624 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:40:58,624 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27092.28 MB 2025-02-14 10:40:58,625 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 10:40:58,625 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 10:40:58,625 - resource_logging.py:150 - __exit__ - DEBUG - Time: 12.83 seconds 2025-02-14 10:40:58,625 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:40:58,625 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15316.98 MB 2025-02-14 10:40:58,625 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27089.43 MB 2025-02-14 10:40:58,625 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11772.45 MB 2025-02-14 10:40:58,625 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54630.81 MB 2025-02-14 10:40:58,625 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32596.03 MB 2025-02-14 10:40:58,625 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22034.78 MB 2025-02-14 10:40:58,625 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27092.28 MB 2025-02-14 10:40:58,892 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 10:40:58,892 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 10:40:58,892 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 10:40:58,892 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:40:58,892 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27089.43 MB 2025-02-14 10:40:58,892 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20321.37 MB 2025-02-14 10:40:58,892 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6768.06 MB 2025-02-14 10:40:58,892 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32596.03 MB 2025-02-14 10:40:58,892 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32596.03 MB 2025-02-14 10:40:58,892 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:40:58,892 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29601.09 MB 2025-02-14 10:40:58,910 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 10:40:58,910 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 10:40:58,916 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 10:40:58,916 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 10:40:58,916 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:40:58,916 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:40:58,916 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20321.37 MB 2025-02-14 10:40:58,916 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28760.39 MB 2025-02-14 10:40:58,917 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 10:40:58,917 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32596.03 MB 2025-02-14 10:40:58,917 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40986.74 MB 2025-02-14 10:40:58,917 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 10:40:58,917 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28760.39 MB 2025-02-14 10:40:59,081 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 10:40:59,083 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:40:59,083 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 10:40:59,084 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:40:59,084 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 10:40:59,088 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 10:40:59,089 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:40:59,089 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 10:40:59,090 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 10:43:22,049 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:43:22,050 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 10:43:22,057 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 10:43:22,065 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:43:22,065 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1048, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 10:43:22,067 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:43:22,067 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1048, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 10:43:38,146 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 10:43:38,146 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 10:43:38,146 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.07 seconds 2025-02-14 10:43:38,146 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:43:38,146 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20271.34 MB 2025-02-14 10:43:38,146 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23981.20 MB 2025-02-14 10:43:38,147 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3709.86 MB 2025-02-14 10:43:38,147 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53571.75 MB 2025-02-14 10:43:38,147 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33187.43 MB 2025-02-14 10:43:38,147 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20384.32 MB 2025-02-14 10:43:38,147 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32913.60 MB 2025-02-14 10:43:38,205 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 10:43:38,205 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 10:43:38,205 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 10:43:38,205 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:43:38,205 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23981.20 MB 2025-02-14 10:43:38,206 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21226.08 MB 2025-02-14 10:43:38,206 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2755.12 MB 2025-02-14 10:43:38,206 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33187.43 MB 2025-02-14 10:43:38,206 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40275.80 MB 2025-02-14 10:43:38,206 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7088.37 MB 2025-02-14 10:43:38,206 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33923.21 MB 2025-02-14 10:43:40,107 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 10:43:40,107 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 10:43:40,107 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.90 seconds 2025-02-14 10:43:40,107 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:43:40,107 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21226.08 MB 2025-02-14 10:43:40,107 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21756.92 MB 2025-02-14 10:43:40,107 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 10:43:40,107 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40275.80 MB 2025-02-14 10:43:40,107 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30893.15 MB 2025-02-14 10:43:40,107 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9382.66 MB 2025-02-14 10:43:40,107 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25736.25 MB 2025-02-14 10:43:40,120 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 10:43:40,120 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 10:43:40,121 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:43:40,121 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:43:40,121 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21756.92 MB 2025-02-14 10:43:40,121 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23646.45 MB 2025-02-14 10:43:40,121 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 10:43:40,121 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30893.15 MB 2025-02-14 10:43:40,121 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30893.15 MB 2025-02-14 10:43:40,121 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:43:40,121 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25063.88 MB 2025-02-14 10:43:40,332 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 10:43:40,332 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 10:43:40,332 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 10:43:40,333 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:43:40,333 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23646.45 MB 2025-02-14 10:43:40,333 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25888.31 MB 2025-02-14 10:43:40,333 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 10:43:40,333 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30893.15 MB 2025-02-14 10:43:40,333 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34196.16 MB 2025-02-14 10:43:40,333 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 10:43:40,333 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31432.59 MB 2025-02-14 10:43:40,333 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 10:43:40,333 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 10:43:40,333 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 10:43:40,333 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:43:40,333 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21756.92 MB 2025-02-14 10:43:40,333 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25888.31 MB 2025-02-14 10:43:40,333 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 10:43:40,333 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30893.15 MB 2025-02-14 10:43:40,333 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34196.16 MB 2025-02-14 10:43:40,333 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 10:43:40,333 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31432.59 MB 2025-02-14 10:43:40,501 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 10:43:40,501 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 10:43:40,501 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 10:43:40,501 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:43:40,501 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27421.85 MB 2025-02-14 10:43:40,501 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28188.85 MB 2025-02-14 10:43:40,501 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 10:43:40,501 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34196.16 MB 2025-02-14 10:43:40,501 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34613.49 MB 2025-02-14 10:43:40,501 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 10:43:40,501 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28896.64 MB 2025-02-14 10:43:40,520 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 10:43:40,520 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 10:43:40,520 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:43:40,520 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:43:40,520 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28601.74 MB 2025-02-14 10:43:40,520 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28833.33 MB 2025-02-14 10:43:40,520 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 231.59 MB 2025-02-14 10:43:40,520 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34613.49 MB 2025-02-14 10:43:40,520 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34613.49 MB 2025-02-14 10:43:40,520 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:43:40,520 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29020.09 MB 2025-02-14 10:43:40,522 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 10:43:40,522 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 10:43:40,522 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.45 seconds 2025-02-14 10:43:40,522 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:43:40,522 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16620.02 MB 2025-02-14 10:43:40,522 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29034.40 MB 2025-02-14 10:43:40,522 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12414.38 MB 2025-02-14 10:43:40,522 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53571.75 MB 2025-02-14 10:43:40,522 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34613.49 MB 2025-02-14 10:43:40,522 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18958.25 MB 2025-02-14 10:43:40,522 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29034.40 MB 2025-02-14 10:43:40,789 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 10:43:40,789 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 10:43:40,789 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 10:43:40,789 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:43:40,789 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29034.40 MB 2025-02-14 10:43:40,789 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21624.41 MB 2025-02-14 10:43:40,789 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7409.99 MB 2025-02-14 10:43:40,789 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34613.49 MB 2025-02-14 10:43:40,789 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34613.49 MB 2025-02-14 10:43:40,789 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:43:40,789 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31546.07 MB 2025-02-14 10:43:40,807 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 10:43:40,807 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 10:43:40,813 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 10:43:40,813 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 10:43:40,813 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:43:40,813 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:43:40,813 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21624.41 MB 2025-02-14 10:43:40,813 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30063.43 MB 2025-02-14 10:43:40,813 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 10:43:40,813 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34613.49 MB 2025-02-14 10:43:40,813 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43004.20 MB 2025-02-14 10:43:40,814 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 10:43:40,814 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30063.43 MB 2025-02-14 10:43:40,978 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 10:43:40,979 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:43:40,979 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 10:43:40,980 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:43:40,980 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 10:43:40,985 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 10:43:40,986 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:43:40,986 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 10:43:40,986 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 10:43:50,518 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:43:50,518 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 10:43:50,523 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 10:43:50,526 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:43:50,526 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 3216, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 10:43:50,527 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:43:50,527 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 3216, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 10:44:40,602 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 10:44:40,603 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 10:44:40,603 - resource_logging.py:150 - __exit__ - DEBUG - Time: 50.06 seconds 2025-02-14 10:44:40,603 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:44:40,603 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35380.41 MB 2025-02-14 10:44:40,603 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46761.65 MB 2025-02-14 10:44:40,603 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11381.24 MB 2025-02-14 10:44:40,603 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 78003.57 MB 2025-02-14 10:44:40,603 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50704.94 MB 2025-02-14 10:44:40,603 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27298.63 MB 2025-02-14 10:44:40,603 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 58142.89 MB 2025-02-14 10:44:40,829 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 10:44:40,829 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 10:44:40,829 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 10:44:40,829 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:44:40,829 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46761.65 MB 2025-02-14 10:44:40,829 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32497.88 MB 2025-02-14 10:44:40,829 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -14263.77 MB 2025-02-14 10:44:40,829 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50704.94 MB 2025-02-14 10:44:40,829 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 94264.89 MB 2025-02-14 10:44:40,829 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 43559.94 MB 2025-02-14 10:44:40,829 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 79869.80 MB 2025-02-14 10:44:42,830 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 10:44:42,830 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 10:44:42,830 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.00 seconds 2025-02-14 10:44:42,830 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:44:42,830 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32497.88 MB 2025-02-14 10:44:42,830 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33028.72 MB 2025-02-14 10:44:42,830 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 10:44:42,830 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 94264.89 MB 2025-02-14 10:44:42,830 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35047.60 MB 2025-02-14 10:44:42,830 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -59217.28 MB 2025-02-14 10:44:42,830 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37009.09 MB 2025-02-14 10:44:42,849 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 10:44:42,849 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 10:44:42,849 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:44:42,849 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:44:42,849 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33028.72 MB 2025-02-14 10:44:42,849 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34918.26 MB 2025-02-14 10:44:42,849 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 10:44:42,849 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35047.60 MB 2025-02-14 10:44:42,849 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38350.62 MB 2025-02-14 10:44:42,849 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 10:44:42,849 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36335.68 MB 2025-02-14 10:44:43,058 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 10:44:43,058 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 10:44:43,058 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 10:44:43,058 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:44:43,058 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34918.26 MB 2025-02-14 10:44:43,058 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37160.11 MB 2025-02-14 10:44:43,058 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 10:44:43,058 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38350.62 MB 2025-02-14 10:44:43,058 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44956.65 MB 2025-02-14 10:44:43,058 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 10:44:43,058 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42704.39 MB 2025-02-14 10:44:43,059 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 10:44:43,059 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 10:44:43,059 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 10:44:43,059 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:44:43,059 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33028.72 MB 2025-02-14 10:44:43,059 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37160.11 MB 2025-02-14 10:44:43,059 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 10:44:43,059 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35047.60 MB 2025-02-14 10:44:43,059 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44956.65 MB 2025-02-14 10:44:43,059 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9909.04 MB 2025-02-14 10:44:43,059 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42704.39 MB 2025-02-14 10:44:43,228 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 10:44:43,228 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 10:44:43,228 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 10:44:43,228 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:44:43,228 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38693.65 MB 2025-02-14 10:44:43,228 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39460.66 MB 2025-02-14 10:44:43,228 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 10:44:43,228 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44956.65 MB 2025-02-14 10:44:43,228 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45371.88 MB 2025-02-14 10:44:43,228 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 10:44:43,228 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40168.44 MB 2025-02-14 10:44:43,247 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 10:44:43,247 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 10:44:43,247 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:44:43,247 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:44:43,247 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39873.54 MB 2025-02-14 10:44:43,247 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40102.27 MB 2025-02-14 10:44:43,247 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.73 MB 2025-02-14 10:44:43,247 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45371.88 MB 2025-02-14 10:44:43,247 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45371.88 MB 2025-02-14 10:44:43,247 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:44:43,247 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40305.74 MB 2025-02-14 10:44:43,248 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 10:44:43,248 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 10:44:43,248 - resource_logging.py:150 - __exit__ - DEBUG - Time: 52.72 seconds 2025-02-14 10:44:43,248 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:44:43,248 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24174.56 MB 2025-02-14 10:44:43,248 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40303.35 MB 2025-02-14 10:44:43,248 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 16128.79 MB 2025-02-14 10:44:43,248 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 66796.39 MB 2025-02-14 10:44:43,248 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45371.88 MB 2025-02-14 10:44:43,248 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21424.50 MB 2025-02-14 10:44:43,248 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40305.74 MB 2025-02-14 10:44:43,517 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 10:44:43,517 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 10:44:43,517 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 10:44:43,517 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:44:43,517 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40303.35 MB 2025-02-14 10:44:43,517 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29178.95 MB 2025-02-14 10:44:43,517 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -11124.40 MB 2025-02-14 10:44:43,517 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45371.88 MB 2025-02-14 10:44:43,517 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45371.88 MB 2025-02-14 10:44:43,517 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:44:43,517 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42815.01 MB 2025-02-14 10:44:43,535 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 10:44:43,536 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 10:44:43,542 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 10:44:43,542 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 10:44:43,542 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:44:43,542 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:44:43,542 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29178.95 MB 2025-02-14 10:44:43,542 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37617.64 MB 2025-02-14 10:44:43,542 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8438.69 MB 2025-02-14 10:44:43,542 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45371.88 MB 2025-02-14 10:44:43,542 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49568.28 MB 2025-02-14 10:44:43,542 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4196.40 MB 2025-02-14 10:44:43,542 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37617.64 MB 2025-02-14 10:44:43,704 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 10:44:43,706 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:44:43,706 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 10:44:43,707 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:44:43,707 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 10:44:43,711 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 10:44:43,712 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:44:43,712 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 10:44:43,713 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 10:45:37,527 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:45:37,527 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 10:45:37,532 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 10:45:37,536 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:45:37,536 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 181, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 10:45:37,537 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:45:37,537 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 181, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 10:45:40,342 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 10:45:40,342 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 10:45:40,342 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.80 seconds 2025-02-14 10:45:40,342 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:45:40,342 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14229.94 MB 2025-02-14 10:45:40,342 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14870.49 MB 2025-02-14 10:45:40,342 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 640.55 MB 2025-02-14 10:45:40,342 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57956.89 MB 2025-02-14 10:45:40,343 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17853.05 MB 2025-02-14 10:45:40,343 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -40103.84 MB 2025-02-14 10:45:40,343 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23702.12 MB 2025-02-14 10:45:40,356 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 10:45:40,356 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 10:45:40,356 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:45:40,356 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:45:40,356 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14870.49 MB 2025-02-14 10:45:40,356 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15082.91 MB 2025-02-14 10:45:40,356 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 212.41 MB 2025-02-14 10:45:40,356 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17853.05 MB 2025-02-14 10:45:40,356 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18444.45 MB 2025-02-14 10:45:40,356 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 591.40 MB 2025-02-14 10:45:40,356 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17234.33 MB 2025-02-14 10:45:41,164 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 10:45:41,165 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 10:45:41,165 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.81 seconds 2025-02-14 10:45:41,165 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:45:41,165 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15082.91 MB 2025-02-14 10:45:41,165 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15304.53 MB 2025-02-14 10:45:41,165 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 221.63 MB 2025-02-14 10:45:41,165 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18444.45 MB 2025-02-14 10:45:41,165 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17972.59 MB 2025-02-14 10:45:41,165 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -471.86 MB 2025-02-14 10:45:41,165 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19254.38 MB 2025-02-14 10:45:41,173 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 10:45:41,173 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 10:45:41,173 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 10:45:41,173 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:45:41,173 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15304.47 MB 2025-02-14 10:45:41,173 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16093.16 MB 2025-02-14 10:45:41,173 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 788.69 MB 2025-02-14 10:45:41,173 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17972.59 MB 2025-02-14 10:45:41,173 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17972.59 MB 2025-02-14 10:45:41,173 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:45:41,173 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16684.94 MB 2025-02-14 10:45:41,265 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 10:45:41,265 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 10:45:41,265 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 10:45:41,266 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:45:41,266 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16093.16 MB 2025-02-14 10:45:41,266 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17029.17 MB 2025-02-14 10:45:41,266 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 936.01 MB 2025-02-14 10:45:41,266 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17972.59 MB 2025-02-14 10:45:41,266 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20338.18 MB 2025-02-14 10:45:41,266 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2365.59 MB 2025-02-14 10:45:41,266 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19345.18 MB 2025-02-14 10:45:41,266 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 10:45:41,266 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 10:45:41,266 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 10:45:41,266 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:45:41,266 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15304.47 MB 2025-02-14 10:45:41,266 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17029.17 MB 2025-02-14 10:45:41,266 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1724.70 MB 2025-02-14 10:45:41,266 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17972.59 MB 2025-02-14 10:45:41,266 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20338.18 MB 2025-02-14 10:45:41,266 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2365.59 MB 2025-02-14 10:45:41,266 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19345.18 MB 2025-02-14 10:45:41,338 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 10:45:41,338 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 10:45:41,338 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 10:45:41,338 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:45:41,338 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17669.42 MB 2025-02-14 10:45:41,338 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17989.91 MB 2025-02-14 10:45:41,338 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 320.49 MB 2025-02-14 10:45:41,338 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20338.18 MB 2025-02-14 10:45:41,338 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20508.05 MB 2025-02-14 10:45:41,338 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 169.87 MB 2025-02-14 10:45:41,338 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18293.32 MB 2025-02-14 10:45:41,348 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 10:45:41,348 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 10:45:41,348 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:45:41,348 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:45:41,348 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18162.30 MB 2025-02-14 10:45:41,348 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18385.92 MB 2025-02-14 10:45:41,348 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 223.62 MB 2025-02-14 10:45:41,348 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20508.05 MB 2025-02-14 10:45:41,348 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20508.05 MB 2025-02-14 10:45:41,348 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:45:41,348 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18405.45 MB 2025-02-14 10:45:41,349 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 10:45:41,349 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 10:45:41,349 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.81 seconds 2025-02-14 10:45:41,349 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:45:41,349 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13599.32 MB 2025-02-14 10:45:41,349 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18586.50 MB 2025-02-14 10:45:41,349 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4987.17 MB 2025-02-14 10:45:41,349 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57956.89 MB 2025-02-14 10:45:41,349 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20508.05 MB 2025-02-14 10:45:41,349 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -37448.84 MB 2025-02-14 10:45:41,349 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18586.50 MB 2025-02-14 10:45:41,615 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 10:45:41,615 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 10:45:41,615 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 10:45:41,615 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:45:41,615 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18586.50 MB 2025-02-14 10:45:41,615 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17496.76 MB 2025-02-14 10:45:41,615 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1089.74 MB 2025-02-14 10:45:41,615 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20508.05 MB 2025-02-14 10:45:41,615 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20508.05 MB 2025-02-14 10:45:41,615 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:45:41,615 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19187.82 MB 2025-02-14 10:45:41,633 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8142, cut from 8144 2025-02-14 10:45:41,633 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 10:45:41,639 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 10:45:41,639 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 10:45:41,639 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:45:41,639 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:45:41,639 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17496.76 MB 2025-02-14 10:45:41,639 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25914.92 MB 2025-02-14 10:45:41,639 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8418.15 MB 2025-02-14 10:45:41,639 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20508.05 MB 2025-02-14 10:45:41,639 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30970.74 MB 2025-02-14 10:45:41,639 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10462.69 MB 2025-02-14 10:45:41,639 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25914.92 MB 2025-02-14 10:45:41,803 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7934] 2025-02-14 10:45:41,804 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:45:41,804 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 10:45:41,805 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:45:41,805 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 10:45:41,810 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 10:45:41,811 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:45:41,811 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 10:45:41,811 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 10:46:44,920 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:46:44,920 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 10:46:44,927 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 10:46:44,933 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:46:44,933 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1186, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 10:46:44,935 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:46:44,935 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1186, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 10:47:03,193 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 10:47:03,193 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 10:47:03,193 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.25 seconds 2025-02-14 10:47:03,193 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:47:03,193 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21232.94 MB 2025-02-14 10:47:03,193 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25430.13 MB 2025-02-14 10:47:03,193 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4197.19 MB 2025-02-14 10:47:03,193 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43524.29 MB 2025-02-14 10:47:03,193 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29475.47 MB 2025-02-14 10:47:03,193 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14048.82 MB 2025-02-14 10:47:03,193 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34328.19 MB 2025-02-14 10:47:03,274 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 10:47:03,274 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 10:47:03,274 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 10:47:03,274 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:47:03,274 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25430.13 MB 2025-02-14 10:47:03,274 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21943.49 MB 2025-02-14 10:47:03,274 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3486.64 MB 2025-02-14 10:47:03,274 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29475.47 MB 2025-02-14 10:47:03,274 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45329.94 MB 2025-02-14 10:47:03,274 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 15854.47 MB 2025-02-14 10:47:03,274 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37985.59 MB 2025-02-14 10:47:05,213 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 10:47:05,213 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 10:47:05,213 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-14 10:47:05,213 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:47:05,213 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21943.49 MB 2025-02-14 10:47:05,213 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22474.34 MB 2025-02-14 10:47:05,214 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 10:47:05,214 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45329.94 MB 2025-02-14 10:47:05,214 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26692.55 MB 2025-02-14 10:47:05,214 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18637.39 MB 2025-02-14 10:47:05,214 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26454.71 MB 2025-02-14 10:47:05,233 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 10:47:05,233 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 10:47:05,233 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:47:05,233 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:47:05,233 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22474.34 MB 2025-02-14 10:47:05,233 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24363.87 MB 2025-02-14 10:47:05,233 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 10:47:05,233 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26692.55 MB 2025-02-14 10:47:05,233 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28579.99 MB 2025-02-14 10:47:05,233 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 10:47:05,233 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25781.30 MB 2025-02-14 10:47:05,448 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 10:47:05,448 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 10:47:05,448 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 10:47:05,448 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:47:05,448 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24363.87 MB 2025-02-14 10:47:05,448 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26605.73 MB 2025-02-14 10:47:05,448 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 10:47:05,448 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28579.99 MB 2025-02-14 10:47:05,448 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34242.30 MB 2025-02-14 10:47:05,448 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 10:47:05,448 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32150.01 MB 2025-02-14 10:47:05,449 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 10:47:05,449 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 10:47:05,449 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 10:47:05,449 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:47:05,449 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22474.34 MB 2025-02-14 10:47:05,449 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26605.73 MB 2025-02-14 10:47:05,449 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 10:47:05,449 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26692.55 MB 2025-02-14 10:47:05,449 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34242.30 MB 2025-02-14 10:47:05,449 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-14 10:47:05,449 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32150.01 MB 2025-02-14 10:47:05,617 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 10:47:05,618 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 10:47:05,618 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 10:47:05,618 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:47:05,618 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28139.27 MB 2025-02-14 10:47:05,618 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28906.27 MB 2025-02-14 10:47:05,618 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 10:47:05,618 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34242.30 MB 2025-02-14 10:47:05,618 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34657.53 MB 2025-02-14 10:47:05,618 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 10:47:05,618 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29614.06 MB 2025-02-14 10:47:05,637 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 10:47:05,637 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 10:47:05,637 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:47:05,637 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:47:05,637 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29319.16 MB 2025-02-14 10:47:05,637 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29544.95 MB 2025-02-14 10:47:05,637 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 225.79 MB 2025-02-14 10:47:05,637 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34657.53 MB 2025-02-14 10:47:05,637 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34657.53 MB 2025-02-14 10:47:05,637 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:47:05,637 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29779.11 MB 2025-02-14 10:47:05,638 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 10:47:05,638 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 10:47:05,638 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.70 seconds 2025-02-14 10:47:05,638 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:47:05,638 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17100.83 MB 2025-02-14 10:47:05,638 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29744.97 MB 2025-02-14 10:47:05,638 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12644.14 MB 2025-02-14 10:47:05,638 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43524.29 MB 2025-02-14 10:47:05,638 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34657.53 MB 2025-02-14 10:47:05,638 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8866.76 MB 2025-02-14 10:47:05,638 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29779.11 MB 2025-02-14 10:47:05,905 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 10:47:05,905 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 10:47:05,905 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 10:47:05,905 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:47:05,905 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29744.97 MB 2025-02-14 10:47:05,905 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22088.83 MB 2025-02-14 10:47:05,905 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7656.13 MB 2025-02-14 10:47:05,905 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34657.53 MB 2025-02-14 10:47:05,905 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34657.53 MB 2025-02-14 10:47:05,905 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:47:05,905 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32243.42 MB 2025-02-14 10:47:05,922 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8119, cut from 8121 2025-02-14 10:47:05,923 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 10:47:05,928 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 10:47:05,928 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 10:47:05,928 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:47:05,928 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:47:05,928 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22088.83 MB 2025-02-14 10:47:05,928 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30484.05 MB 2025-02-14 10:47:05,928 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8395.21 MB 2025-02-14 10:47:05,928 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34657.53 MB 2025-02-14 10:47:05,928 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38830.87 MB 2025-02-14 10:47:05,928 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4173.33 MB 2025-02-14 10:47:05,928 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30484.05 MB 2025-02-14 10:47:06,091 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7911] 2025-02-14 10:47:06,093 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:47:06,093 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 10:47:06,094 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:47:06,094 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 10:47:06,099 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 10:47:06,100 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:47:06,100 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 10:47:06,100 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 10:48:16,692 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:48:16,692 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 10:48:16,697 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 10:48:16,701 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:48:16,701 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1487, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 10:48:16,702 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:48:16,702 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1487, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 10:48:39,600 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 10:48:39,600 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 10:48:39,600 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.89 seconds 2025-02-14 10:48:39,600 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:48:39,600 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23330.36 MB 2025-02-14 10:48:39,600 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28592.77 MB 2025-02-14 10:48:39,600 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5262.41 MB 2025-02-14 10:48:39,600 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47177.53 MB 2025-02-14 10:48:39,600 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38877.00 MB 2025-02-14 10:48:39,600 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8300.53 MB 2025-02-14 10:48:39,600 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37558.88 MB 2025-02-14 10:48:39,681 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 10:48:39,681 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 10:48:39,681 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 10:48:39,681 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:48:39,681 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28592.77 MB 2025-02-14 10:48:39,681 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23508.30 MB 2025-02-14 10:48:39,681 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5084.47 MB 2025-02-14 10:48:39,681 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38877.00 MB 2025-02-14 10:48:39,681 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47938.80 MB 2025-02-14 10:48:39,681 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9061.79 MB 2025-02-14 10:48:39,681 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42272.41 MB 2025-02-14 10:48:41,598 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 10:48:41,598 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 10:48:41,598 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 10:48:41,598 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:48:41,598 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23508.30 MB 2025-02-14 10:48:41,598 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24039.14 MB 2025-02-14 10:48:41,598 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 10:48:41,598 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47938.80 MB 2025-02-14 10:48:41,598 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33613.15 MB 2025-02-14 10:48:41,598 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14325.65 MB 2025-02-14 10:48:41,598 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28018.48 MB 2025-02-14 10:48:41,612 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 10:48:41,612 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 10:48:41,612 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:48:41,612 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:48:41,612 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24039.14 MB 2025-02-14 10:48:41,612 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25928.68 MB 2025-02-14 10:48:41,612 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 10:48:41,612 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33613.15 MB 2025-02-14 10:48:41,612 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33613.15 MB 2025-02-14 10:48:41,612 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:48:41,612 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27346.10 MB 2025-02-14 10:48:41,828 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 10:48:41,828 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 10:48:41,828 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 10:48:41,828 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:48:41,828 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25928.68 MB 2025-02-14 10:48:41,829 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28170.53 MB 2025-02-14 10:48:41,829 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 10:48:41,829 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33613.15 MB 2025-02-14 10:48:41,829 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37388.03 MB 2025-02-14 10:48:41,829 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 10:48:41,829 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33714.81 MB 2025-02-14 10:48:41,829 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 10:48:41,829 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 10:48:41,829 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 10:48:41,829 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:48:41,829 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24039.14 MB 2025-02-14 10:48:41,829 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28170.53 MB 2025-02-14 10:48:41,829 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 10:48:41,829 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33613.15 MB 2025-02-14 10:48:41,829 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37388.03 MB 2025-02-14 10:48:41,829 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 10:48:41,829 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33714.81 MB 2025-02-14 10:48:42,005 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 10:48:42,005 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 10:48:42,005 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 10:48:42,005 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:48:42,005 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29704.07 MB 2025-02-14 10:48:42,005 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30471.08 MB 2025-02-14 10:48:42,005 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 10:48:42,005 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37388.03 MB 2025-02-14 10:48:42,005 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37803.26 MB 2025-02-14 10:48:42,005 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 10:48:42,005 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31178.86 MB 2025-02-14 10:48:42,024 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 10:48:42,024 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 10:48:42,024 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:48:42,024 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:48:42,024 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30883.96 MB 2025-02-14 10:48:42,024 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31112.81 MB 2025-02-14 10:48:42,024 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.85 MB 2025-02-14 10:48:42,024 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37803.26 MB 2025-02-14 10:48:42,024 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37803.26 MB 2025-02-14 10:48:42,024 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:48:42,024 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31317.04 MB 2025-02-14 10:48:42,025 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 10:48:42,025 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 10:48:42,025 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.32 seconds 2025-02-14 10:48:42,025 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:48:42,025 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18149.53 MB 2025-02-14 10:48:42,025 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31312.83 MB 2025-02-14 10:48:42,025 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13163.29 MB 2025-02-14 10:48:42,025 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47177.53 MB 2025-02-14 10:48:42,025 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37803.26 MB 2025-02-14 10:48:42,025 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9374.27 MB 2025-02-14 10:48:42,025 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31317.04 MB 2025-02-14 10:48:42,295 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 10:48:42,295 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 10:48:42,295 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 10:48:42,295 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:48:42,295 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31312.83 MB 2025-02-14 10:48:42,295 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23137.54 MB 2025-02-14 10:48:42,295 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8175.28 MB 2025-02-14 10:48:42,295 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37803.26 MB 2025-02-14 10:48:42,295 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37803.26 MB 2025-02-14 10:48:42,295 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:48:42,295 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33811.28 MB 2025-02-14 10:48:42,313 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8119, cut from 8121 2025-02-14 10:48:42,314 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 10:48:42,320 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 10:48:42,320 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 10:48:42,320 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:48:42,320 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:48:42,320 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23137.54 MB 2025-02-14 10:48:42,320 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31532.76 MB 2025-02-14 10:48:42,320 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8395.21 MB 2025-02-14 10:48:42,320 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37803.26 MB 2025-02-14 10:48:42,320 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46149.93 MB 2025-02-14 10:48:42,320 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8346.66 MB 2025-02-14 10:48:42,320 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31532.76 MB 2025-02-14 10:48:42,489 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7911] 2025-02-14 10:48:42,491 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:48:42,491 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 10:48:42,492 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:48:42,492 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 10:48:42,498 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 10:48:42,499 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:48:42,499 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 10:48:42,499 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 10:49:40,875 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:49:40,875 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 10:49:40,880 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 10:49:40,884 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:49:40,884 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1720, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 10:49:40,885 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:49:40,885 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1720, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 10:50:07,342 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 10:50:07,342 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 10:50:07,342 - resource_logging.py:150 - __exit__ - DEBUG - Time: 26.45 seconds 2025-02-14 10:50:07,342 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:50:07,342 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24953.94 MB 2025-02-14 10:50:07,342 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31041.97 MB 2025-02-14 10:50:07,342 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6088.03 MB 2025-02-14 10:50:07,342 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54496.59 MB 2025-02-14 10:50:07,342 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39701.18 MB 2025-02-14 10:50:07,342 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14795.41 MB 2025-02-14 10:50:07,342 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39861.94 MB 2025-02-14 10:50:07,445 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 10:50:07,446 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 10:50:07,446 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 10:50:07,446 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:50:07,446 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31041.97 MB 2025-02-14 10:50:07,446 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24719.59 MB 2025-02-14 10:50:07,446 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6322.38 MB 2025-02-14 10:50:07,446 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39701.18 MB 2025-02-14 10:50:07,446 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57719.91 MB 2025-02-14 10:50:07,446 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 18018.73 MB 2025-02-14 10:50:07,446 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48795.23 MB 2025-02-14 10:50:09,379 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 10:50:09,379 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 10:50:09,379 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 10:50:09,379 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:50:09,379 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24719.59 MB 2025-02-14 10:50:09,379 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25250.44 MB 2025-02-14 10:50:09,379 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 10:50:09,379 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57719.91 MB 2025-02-14 10:50:09,379 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35028.73 MB 2025-02-14 10:50:09,379 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22691.18 MB 2025-02-14 10:50:09,379 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29229.77 MB 2025-02-14 10:50:09,392 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 10:50:09,392 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 10:50:09,392 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:50:09,392 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:50:09,392 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25250.44 MB 2025-02-14 10:50:09,392 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27139.97 MB 2025-02-14 10:50:09,392 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 10:50:09,392 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35028.73 MB 2025-02-14 10:50:09,392 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35028.73 MB 2025-02-14 10:50:09,392 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:50:09,392 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28557.40 MB 2025-02-14 10:50:09,607 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 10:50:09,607 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 10:50:09,607 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 10:50:09,607 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:50:09,607 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27139.97 MB 2025-02-14 10:50:09,607 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29381.83 MB 2025-02-14 10:50:09,607 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 10:50:09,607 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35028.73 MB 2025-02-14 10:50:09,607 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38331.74 MB 2025-02-14 10:50:09,607 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 10:50:09,607 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34926.11 MB 2025-02-14 10:50:09,607 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 10:50:09,607 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 10:50:09,607 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 10:50:09,607 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:50:09,607 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25250.44 MB 2025-02-14 10:50:09,607 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29381.83 MB 2025-02-14 10:50:09,607 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 10:50:09,607 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35028.73 MB 2025-02-14 10:50:09,607 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38331.74 MB 2025-02-14 10:50:09,607 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 10:50:09,608 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34926.11 MB 2025-02-14 10:50:09,774 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 10:50:09,774 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 10:50:09,774 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 10:50:09,774 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:50:09,774 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30915.37 MB 2025-02-14 10:50:09,774 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31682.37 MB 2025-02-14 10:50:09,774 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 10:50:09,774 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38331.74 MB 2025-02-14 10:50:09,774 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38746.98 MB 2025-02-14 10:50:09,774 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 10:50:09,774 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32390.16 MB 2025-02-14 10:50:09,793 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 10:50:09,793 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 10:50:09,793 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:50:09,793 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:50:09,793 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32095.26 MB 2025-02-14 10:50:09,793 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32323.43 MB 2025-02-14 10:50:09,793 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.17 MB 2025-02-14 10:50:09,793 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38746.98 MB 2025-02-14 10:50:09,793 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38746.98 MB 2025-02-14 10:50:09,793 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:50:09,793 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32535.29 MB 2025-02-14 10:50:09,794 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 10:50:09,794 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 10:50:09,794 - resource_logging.py:150 - __exit__ - DEBUG - Time: 28.91 seconds 2025-02-14 10:50:09,794 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:50:09,794 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18961.32 MB 2025-02-14 10:50:09,794 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32523.42 MB 2025-02-14 10:50:09,794 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13562.10 MB 2025-02-14 10:50:09,794 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54496.59 MB 2025-02-14 10:50:09,794 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38746.98 MB 2025-02-14 10:50:09,794 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15749.61 MB 2025-02-14 10:50:09,794 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32535.29 MB 2025-02-14 10:50:10,062 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 10:50:10,062 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 10:50:10,062 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 10:50:10,062 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:50:10,062 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32523.42 MB 2025-02-14 10:50:10,062 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23948.95 MB 2025-02-14 10:50:10,062 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8574.47 MB 2025-02-14 10:50:10,062 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38746.98 MB 2025-02-14 10:50:10,062 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38746.98 MB 2025-02-14 10:50:10,062 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:50:10,062 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35021.58 MB 2025-02-14 10:50:10,080 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8118, cut from 8120 2025-02-14 10:50:10,080 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 10:50:10,086 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 10:50:10,086 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 10:50:10,086 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:50:10,086 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:50:10,086 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23948.95 MB 2025-02-14 10:50:10,086 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32342.23 MB 2025-02-14 10:50:10,086 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8393.27 MB 2025-02-14 10:50:10,086 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38746.98 MB 2025-02-14 10:50:10,086 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42920.31 MB 2025-02-14 10:50:10,086 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4173.33 MB 2025-02-14 10:50:10,086 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32342.23 MB 2025-02-14 10:50:10,252 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7910] 2025-02-14 10:50:10,253 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:50:10,253 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 10:50:10,254 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:50:10,254 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 10:50:10,259 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 10:50:10,260 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:50:10,260 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 10:50:10,260 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 10:50:35,108 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:50:35,108 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 10:50:35,113 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 10:50:35,117 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:50:35,117 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1150, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 10:50:35,118 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:50:35,118 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1150, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 10:50:52,940 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 10:50:52,940 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 10:50:52,940 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.82 seconds 2025-02-14 10:50:52,940 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:50:52,940 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20982.09 MB 2025-02-14 10:50:52,940 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25052.66 MB 2025-02-14 10:50:52,940 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4070.57 MB 2025-02-14 10:50:52,940 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51266.98 MB 2025-02-14 10:50:52,940 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29328.67 MB 2025-02-14 10:50:52,940 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21938.31 MB 2025-02-14 10:50:52,940 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33851.65 MB 2025-02-14 10:50:53,042 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 10:50:53,042 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 10:50:53,042 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 10:50:53,042 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:50:53,042 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25052.66 MB 2025-02-14 10:50:53,042 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21756.34 MB 2025-02-14 10:50:53,042 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3296.32 MB 2025-02-14 10:50:53,042 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29328.67 MB 2025-02-14 10:50:53,042 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44166.02 MB 2025-02-14 10:50:53,042 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 14837.35 MB 2025-02-14 10:50:53,042 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37278.08 MB 2025-02-14 10:50:55,005 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 10:50:55,006 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 10:50:55,006 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.96 seconds 2025-02-14 10:50:55,006 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:50:55,006 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21756.34 MB 2025-02-14 10:50:55,006 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22287.18 MB 2025-02-14 10:50:55,006 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 10:50:55,006 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44166.02 MB 2025-02-14 10:50:55,006 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26673.68 MB 2025-02-14 10:50:55,006 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17492.34 MB 2025-02-14 10:50:55,006 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26266.52 MB 2025-02-14 10:50:55,019 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 10:50:55,019 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 10:50:55,019 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:50:55,020 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:50:55,020 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22287.18 MB 2025-02-14 10:50:55,020 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24176.72 MB 2025-02-14 10:50:55,020 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 10:50:55,020 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26673.68 MB 2025-02-14 10:50:55,020 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28561.11 MB 2025-02-14 10:50:55,020 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 10:50:55,020 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25594.15 MB 2025-02-14 10:50:55,231 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 10:50:55,231 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 10:50:55,231 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 10:50:55,231 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:50:55,231 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24176.72 MB 2025-02-14 10:50:55,231 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26418.57 MB 2025-02-14 10:50:55,231 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 10:50:55,231 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28561.11 MB 2025-02-14 10:50:55,231 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34223.42 MB 2025-02-14 10:50:55,231 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 10:50:55,231 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31962.86 MB 2025-02-14 10:50:55,232 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 10:50:55,232 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 10:50:55,232 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 10:50:55,232 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:50:55,232 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22287.18 MB 2025-02-14 10:50:55,232 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26418.57 MB 2025-02-14 10:50:55,232 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 10:50:55,232 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26673.68 MB 2025-02-14 10:50:55,232 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34223.42 MB 2025-02-14 10:50:55,232 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-14 10:50:55,232 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31962.86 MB 2025-02-14 10:50:55,399 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 10:50:55,399 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 10:50:55,399 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 10:50:55,399 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:50:55,399 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27952.12 MB 2025-02-14 10:50:55,400 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28719.12 MB 2025-02-14 10:50:55,400 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 10:50:55,400 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34223.42 MB 2025-02-14 10:50:55,400 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34636.56 MB 2025-02-14 10:50:55,400 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 10:50:55,400 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29426.91 MB 2025-02-14 10:50:55,418 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 10:50:55,418 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 10:50:55,418 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:50:55,418 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:50:55,418 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29132.01 MB 2025-02-14 10:50:55,418 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29360.65 MB 2025-02-14 10:50:55,418 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.64 MB 2025-02-14 10:50:55,418 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34636.56 MB 2025-02-14 10:50:55,418 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34636.56 MB 2025-02-14 10:50:55,418 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:50:55,418 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29559.64 MB 2025-02-14 10:50:55,419 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 10:50:55,419 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 10:50:55,419 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.30 seconds 2025-02-14 10:50:55,419 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:50:55,419 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16975.40 MB 2025-02-14 10:50:55,419 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29561.20 MB 2025-02-14 10:50:55,419 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12585.81 MB 2025-02-14 10:50:55,419 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51266.98 MB 2025-02-14 10:50:55,419 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34636.56 MB 2025-02-14 10:50:55,419 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16630.42 MB 2025-02-14 10:50:55,420 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29561.20 MB 2025-02-14 10:50:55,688 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 10:50:55,688 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 10:50:55,688 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 10:50:55,688 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:50:55,688 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29561.20 MB 2025-02-14 10:50:55,688 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21971.79 MB 2025-02-14 10:50:55,688 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7589.42 MB 2025-02-14 10:50:55,688 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34636.56 MB 2025-02-14 10:50:55,688 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34636.56 MB 2025-02-14 10:50:55,688 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:50:55,688 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32066.42 MB 2025-02-14 10:50:55,706 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8141, cut from 8143 2025-02-14 10:50:55,706 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 10:50:55,712 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 10:50:55,712 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 10:50:55,712 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:50:55,712 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:50:55,712 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21971.79 MB 2025-02-14 10:50:55,712 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30389.53 MB 2025-02-14 10:50:55,712 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8417.74 MB 2025-02-14 10:50:55,712 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34636.56 MB 2025-02-14 10:50:55,712 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43004.20 MB 2025-02-14 10:50:55,712 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8367.64 MB 2025-02-14 10:50:55,712 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30389.53 MB 2025-02-14 10:50:55,876 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7933] 2025-02-14 10:50:55,877 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:50:55,878 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 10:50:55,878 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:50:55,878 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 10:50:55,883 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 10:50:55,884 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:50:55,884 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 10:50:55,884 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 10:51:07,394 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:51:07,395 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 10:51:07,400 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 10:51:07,403 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:51:07,403 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 495, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 10:51:07,404 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:51:07,404 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 495, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 10:51:15,089 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 10:51:15,089 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 10:51:15,089 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.68 seconds 2025-02-14 10:51:15,089 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:51:15,089 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16417.95 MB 2025-02-14 10:51:15,089 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18169.72 MB 2025-02-14 10:51:15,089 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1751.78 MB 2025-02-14 10:51:15,089 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51371.84 MB 2025-02-14 10:51:15,089 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22439.53 MB 2025-02-14 10:51:15,089 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28932.31 MB 2025-02-14 10:51:15,089 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27022.59 MB 2025-02-14 10:51:15,124 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 10:51:15,124 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 10:51:15,124 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 10:51:15,124 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:51:15,124 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18169.72 MB 2025-02-14 10:51:15,124 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18351.20 MB 2025-02-14 10:51:15,124 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 181.48 MB 2025-02-14 10:51:15,124 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22439.53 MB 2025-02-14 10:51:15,124 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28630.32 MB 2025-02-14 10:51:15,124 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6190.79 MB 2025-02-14 10:51:15,124 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25805.79 MB 2025-02-14 10:51:17,057 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 10:51:17,057 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 10:51:17,057 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 10:51:17,057 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:51:17,057 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18351.20 MB 2025-02-14 10:51:17,057 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18882.04 MB 2025-02-14 10:51:17,057 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 10:51:17,057 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28630.32 MB 2025-02-14 10:51:17,057 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21158.17 MB 2025-02-14 10:51:17,057 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7472.15 MB 2025-02-14 10:51:17,057 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22862.41 MB 2025-02-14 10:51:17,071 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 10:51:17,071 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 10:51:17,071 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:51:17,071 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:51:17,071 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18882.04 MB 2025-02-14 10:51:17,071 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20771.58 MB 2025-02-14 10:51:17,071 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 10:51:17,071 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21158.17 MB 2025-02-14 10:51:17,071 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24461.18 MB 2025-02-14 10:51:17,071 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 10:51:17,071 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22189.01 MB 2025-02-14 10:51:17,283 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 10:51:17,283 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 10:51:17,283 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 10:51:17,283 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:51:17,283 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20771.58 MB 2025-02-14 10:51:17,283 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23013.43 MB 2025-02-14 10:51:17,283 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 10:51:17,283 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24461.18 MB 2025-02-14 10:51:17,283 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31067.21 MB 2025-02-14 10:51:17,283 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 10:51:17,283 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28557.71 MB 2025-02-14 10:51:17,284 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 10:51:17,284 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 10:51:17,284 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 10:51:17,284 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:51:17,284 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18882.04 MB 2025-02-14 10:51:17,284 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23013.43 MB 2025-02-14 10:51:17,284 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 10:51:17,284 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21158.17 MB 2025-02-14 10:51:17,284 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31067.21 MB 2025-02-14 10:51:17,284 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9909.04 MB 2025-02-14 10:51:17,284 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28557.71 MB 2025-02-14 10:51:17,450 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 10:51:17,450 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 10:51:17,450 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 10:51:17,450 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:51:17,450 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24546.98 MB 2025-02-14 10:51:17,450 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25313.98 MB 2025-02-14 10:51:17,450 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 10:51:17,450 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31067.21 MB 2025-02-14 10:51:17,450 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31480.35 MB 2025-02-14 10:51:17,450 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 10:51:17,450 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26021.77 MB 2025-02-14 10:51:17,468 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 10:51:17,468 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 10:51:17,468 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:51:17,468 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:51:17,468 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25726.87 MB 2025-02-14 10:51:17,468 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25955.75 MB 2025-02-14 10:51:17,468 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.89 MB 2025-02-14 10:51:17,468 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31480.35 MB 2025-02-14 10:51:17,468 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31480.35 MB 2025-02-14 10:51:17,468 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:51:17,468 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26136.19 MB 2025-02-14 10:51:17,469 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 10:51:17,469 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 10:51:17,470 - resource_logging.py:150 - __exit__ - DEBUG - Time: 10.06 seconds 2025-02-14 10:51:17,470 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:51:17,470 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14693.33 MB 2025-02-14 10:51:17,470 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26156.83 MB 2025-02-14 10:51:17,470 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11463.50 MB 2025-02-14 10:51:17,470 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51371.84 MB 2025-02-14 10:51:17,470 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31480.35 MB 2025-02-14 10:51:17,470 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19891.49 MB 2025-02-14 10:51:17,470 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26156.83 MB 2025-02-14 10:51:17,738 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 10:51:17,738 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 10:51:17,738 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 10:51:17,738 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:51:17,738 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26156.83 MB 2025-02-14 10:51:17,738 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19697.72 MB 2025-02-14 10:51:17,738 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6459.11 MB 2025-02-14 10:51:17,738 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31480.35 MB 2025-02-14 10:51:17,738 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31480.35 MB 2025-02-14 10:51:17,738 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:51:17,738 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28668.49 MB 2025-02-14 10:51:17,756 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 10:51:17,756 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 10:51:17,762 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 10:51:17,762 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 10:51:17,762 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:51:17,762 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:51:17,762 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19697.72 MB 2025-02-14 10:51:17,763 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28136.74 MB 2025-02-14 10:51:17,763 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 10:51:17,763 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31480.35 MB 2025-02-14 10:51:17,763 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41970.30 MB 2025-02-14 10:51:17,763 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 10:51:17,763 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28136.74 MB 2025-02-14 10:51:17,927 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 10:51:17,928 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:51:17,928 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 10:51:17,929 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:51:17,929 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 10:51:17,934 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 10:51:17,935 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:51:17,935 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 10:51:17,935 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 10:52:09,226 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:52:09,226 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 10:52:09,231 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 10:52:09,235 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:52:09,235 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 207, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 10:52:09,236 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:52:09,236 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 207, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 10:52:12,462 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 10:52:12,462 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 10:52:12,462 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.22 seconds 2025-02-14 10:52:12,462 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:52:12,462 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14411.12 MB 2025-02-14 10:52:12,462 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15143.68 MB 2025-02-14 10:52:12,462 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 732.56 MB 2025-02-14 10:52:12,462 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54555.31 MB 2025-02-14 10:52:12,462 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17855.15 MB 2025-02-14 10:52:12,462 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -36700.16 MB 2025-02-14 10:52:12,462 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24109.79 MB 2025-02-14 10:52:12,477 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 10:52:12,477 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 10:52:12,477 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:52:12,477 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:52:12,477 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15143.68 MB 2025-02-14 10:52:12,477 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15330.44 MB 2025-02-14 10:52:12,477 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 186.76 MB 2025-02-14 10:52:12,477 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17855.15 MB 2025-02-14 10:52:12,477 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19151.19 MB 2025-02-14 10:52:12,477 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1296.04 MB 2025-02-14 10:52:12,477 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17768.04 MB 2025-02-14 10:52:13,360 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 10:52:13,360 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 10:52:13,360 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.88 seconds 2025-02-14 10:52:13,360 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:52:13,360 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15330.44 MB 2025-02-14 10:52:13,360 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15573.30 MB 2025-02-14 10:52:13,360 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 242.86 MB 2025-02-14 10:52:13,360 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19151.19 MB 2025-02-14 10:52:13,360 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18031.31 MB 2025-02-14 10:52:13,360 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1119.88 MB 2025-02-14 10:52:13,360 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19501.92 MB 2025-02-14 10:52:13,369 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 10:52:13,369 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 10:52:13,369 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:52:13,369 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:52:13,369 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15573.24 MB 2025-02-14 10:52:13,369 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16437.75 MB 2025-02-14 10:52:13,369 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 864.51 MB 2025-02-14 10:52:13,369 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18031.31 MB 2025-02-14 10:52:13,369 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18463.33 MB 2025-02-14 10:52:13,369 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 432.01 MB 2025-02-14 10:52:13,369 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17086.23 MB 2025-02-14 10:52:13,471 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 10:52:13,471 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 10:52:13,471 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 10:52:13,471 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:52:13,471 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16437.75 MB 2025-02-14 10:52:13,471 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17463.43 MB 2025-02-14 10:52:13,471 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1025.68 MB 2025-02-14 10:52:13,471 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18463.33 MB 2025-02-14 10:52:13,471 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21055.41 MB 2025-02-14 10:52:13,471 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2592.08 MB 2025-02-14 10:52:13,471 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20001.09 MB 2025-02-14 10:52:13,472 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 10:52:13,472 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 10:52:13,472 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 10:52:13,472 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:52:13,472 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15573.24 MB 2025-02-14 10:52:13,472 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17463.43 MB 2025-02-14 10:52:13,472 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1890.20 MB 2025-02-14 10:52:13,472 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18031.31 MB 2025-02-14 10:52:13,472 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21055.41 MB 2025-02-14 10:52:13,472 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3024.09 MB 2025-02-14 10:52:13,472 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20001.09 MB 2025-02-14 10:52:13,552 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 10:52:13,552 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 10:52:13,552 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 10:52:13,552 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:52:13,552 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18165.03 MB 2025-02-14 10:52:13,552 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18516.06 MB 2025-02-14 10:52:13,552 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 351.03 MB 2025-02-14 10:52:13,552 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21055.41 MB 2025-02-14 10:52:13,552 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21239.96 MB 2025-02-14 10:52:13,552 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 184.55 MB 2025-02-14 10:52:13,552 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18846.17 MB 2025-02-14 10:52:13,566 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 10:52:13,566 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 10:52:13,566 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:52:13,566 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:52:13,566 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18704.97 MB 2025-02-14 10:52:13,566 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18930.57 MB 2025-02-14 10:52:13,566 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 225.60 MB 2025-02-14 10:52:13,566 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21239.96 MB 2025-02-14 10:52:13,566 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21239.96 MB 2025-02-14 10:52:13,566 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:52:13,566 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18963.63 MB 2025-02-14 10:52:13,567 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 10:52:13,567 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 10:52:13,567 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.33 seconds 2025-02-14 10:52:13,567 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:52:13,567 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13689.91 MB 2025-02-14 10:52:13,567 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19131.25 MB 2025-02-14 10:52:13,567 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5441.34 MB 2025-02-14 10:52:13,567 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54555.31 MB 2025-02-14 10:52:13,567 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21239.96 MB 2025-02-14 10:52:13,567 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33315.36 MB 2025-02-14 10:52:13,568 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19131.25 MB 2025-02-14 10:52:13,833 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 10:52:13,833 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 10:52:13,833 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 10:52:13,833 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:52:13,833 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19131.25 MB 2025-02-14 10:52:13,833 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17664.25 MB 2025-02-14 10:52:13,833 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1467.00 MB 2025-02-14 10:52:13,833 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21239.96 MB 2025-02-14 10:52:13,833 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21239.96 MB 2025-02-14 10:52:13,833 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:52:13,833 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19131.26 MB 2025-02-14 10:52:13,851 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8146, cut from 8148 2025-02-14 10:52:13,851 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 10:52:13,857 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 10:52:13,857 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 10:52:13,858 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:52:13,858 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:52:13,858 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17664.25 MB 2025-02-14 10:52:13,858 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26086.58 MB 2025-02-14 10:52:13,858 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8422.33 MB 2025-02-14 10:52:13,858 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21239.96 MB 2025-02-14 10:52:13,858 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31708.94 MB 2025-02-14 10:52:13,858 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10468.98 MB 2025-02-14 10:52:13,858 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26086.58 MB 2025-02-14 10:52:14,019 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7938] 2025-02-14 10:52:14,020 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:52:14,020 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 10:52:14,021 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:52:14,021 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 10:52:14,026 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 10:52:14,027 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:52:14,027 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 10:52:14,027 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 10:52:29,760 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:52:29,761 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 10:52:29,765 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 10:52:29,769 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:52:29,769 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1255, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 10:52:29,770 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:52:29,770 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1255, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 10:52:49,183 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 10:52:49,183 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 10:52:49,183 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.41 seconds 2025-02-14 10:52:49,183 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:52:49,183 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21713.75 MB 2025-02-14 10:52:49,183 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26155.51 MB 2025-02-14 10:52:49,183 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4441.77 MB 2025-02-14 10:52:49,183 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44268.78 MB 2025-02-14 10:52:49,183 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38094.77 MB 2025-02-14 10:52:49,183 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6174.02 MB 2025-02-14 10:52:49,183 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35035.49 MB 2025-02-14 10:52:49,259 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 10:52:49,259 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 10:52:49,259 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 10:52:49,259 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:52:49,259 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26155.51 MB 2025-02-14 10:52:49,259 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22302.20 MB 2025-02-14 10:52:49,259 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3853.31 MB 2025-02-14 10:52:49,259 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38094.77 MB 2025-02-14 10:52:49,259 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46785.36 MB 2025-02-14 10:52:49,259 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8690.60 MB 2025-02-14 10:52:49,259 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39246.73 MB 2025-02-14 10:52:51,181 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 10:52:51,181 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 10:52:51,181 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 10:52:51,181 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:52:51,181 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22302.20 MB 2025-02-14 10:52:51,181 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22833.05 MB 2025-02-14 10:52:51,181 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 10:52:51,181 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46785.36 MB 2025-02-14 10:52:51,181 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33653.00 MB 2025-02-14 10:52:51,181 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13132.37 MB 2025-02-14 10:52:51,182 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26812.38 MB 2025-02-14 10:52:51,196 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 10:52:51,196 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 10:52:51,196 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:52:51,196 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:52:51,196 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22833.05 MB 2025-02-14 10:52:51,196 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24722.58 MB 2025-02-14 10:52:51,196 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 10:52:51,196 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33653.00 MB 2025-02-14 10:52:51,196 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33653.00 MB 2025-02-14 10:52:51,196 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:52:51,196 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26140.01 MB 2025-02-14 10:52:51,411 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 10:52:51,411 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 10:52:51,411 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 10:52:51,411 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:52:51,411 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24722.58 MB 2025-02-14 10:52:51,411 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26964.44 MB 2025-02-14 10:52:51,411 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 10:52:51,411 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33653.00 MB 2025-02-14 10:52:51,411 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35540.43 MB 2025-02-14 10:52:51,411 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 10:52:51,411 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32508.72 MB 2025-02-14 10:52:51,411 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 10:52:51,411 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 10:52:51,411 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 10:52:51,411 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:52:51,412 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22833.05 MB 2025-02-14 10:52:51,412 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26964.44 MB 2025-02-14 10:52:51,412 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 10:52:51,412 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33653.00 MB 2025-02-14 10:52:51,412 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35540.43 MB 2025-02-14 10:52:51,412 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 10:52:51,412 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32508.72 MB 2025-02-14 10:52:51,590 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 10:52:51,590 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 10:52:51,590 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 10:52:51,590 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:52:51,590 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28497.98 MB 2025-02-14 10:52:51,590 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29264.98 MB 2025-02-14 10:52:51,590 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 10:52:51,590 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35540.43 MB 2025-02-14 10:52:51,590 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35957.77 MB 2025-02-14 10:52:51,590 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 10:52:51,590 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29972.77 MB 2025-02-14 10:52:51,609 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 10:52:51,609 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 10:52:51,610 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:52:51,610 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:52:51,610 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29677.87 MB 2025-02-14 10:52:51,610 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29906.43 MB 2025-02-14 10:52:51,610 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.56 MB 2025-02-14 10:52:51,610 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35957.77 MB 2025-02-14 10:52:51,610 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35957.77 MB 2025-02-14 10:52:51,610 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:52:51,610 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30143.38 MB 2025-02-14 10:52:51,611 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 10:52:51,611 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 10:52:51,611 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.84 seconds 2025-02-14 10:52:51,611 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:52:51,611 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17341.23 MB 2025-02-14 10:52:51,611 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30106.60 MB 2025-02-14 10:52:51,611 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12765.37 MB 2025-02-14 10:52:51,611 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44268.78 MB 2025-02-14 10:52:51,611 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35957.77 MB 2025-02-14 10:52:51,611 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8311.01 MB 2025-02-14 10:52:51,611 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30143.38 MB 2025-02-14 10:52:51,881 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 10:52:51,881 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 10:52:51,881 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 10:52:51,881 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:52:51,881 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30106.60 MB 2025-02-14 10:52:51,881 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22331.52 MB 2025-02-14 10:52:51,881 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7775.07 MB 2025-02-14 10:52:51,881 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35957.77 MB 2025-02-14 10:52:51,881 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35957.77 MB 2025-02-14 10:52:51,881 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:52:51,881 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32606.90 MB 2025-02-14 10:52:51,898 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8125, cut from 8127 2025-02-14 10:52:51,899 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 10:52:51,905 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 10:52:51,905 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 10:52:51,905 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:52:51,905 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:52:51,905 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22331.52 MB 2025-02-14 10:52:51,905 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30732.46 MB 2025-02-14 10:52:51,905 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8400.94 MB 2025-02-14 10:52:51,905 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35957.77 MB 2025-02-14 10:52:51,905 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40133.20 MB 2025-02-14 10:52:51,905 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4175.43 MB 2025-02-14 10:52:51,905 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30732.46 MB 2025-02-14 10:52:52,073 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7917] 2025-02-14 10:52:52,075 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:52:52,075 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 10:52:52,076 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:52:52,076 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 10:52:52,081 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 10:52:52,082 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:52:52,082 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 10:52:52,082 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 10:54:13,356 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:54:13,356 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 10:54:13,362 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 10:54:13,366 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:54:13,366 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 307, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 10:54:13,367 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:54:13,367 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 307, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 10:54:18,075 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 10:54:18,075 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 10:54:18,075 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.70 seconds 2025-02-14 10:54:18,075 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:54:18,075 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15107.93 MB 2025-02-14 10:54:18,075 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16194.39 MB 2025-02-14 10:54:18,075 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1086.46 MB 2025-02-14 10:54:18,075 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52661.58 MB 2025-02-14 10:54:18,075 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17995.66 MB 2025-02-14 10:54:18,075 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34665.92 MB 2025-02-14 10:54:18,075 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25033.09 MB 2025-02-14 10:54:18,097 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 10:54:18,097 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 10:54:18,097 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:54:18,097 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:54:18,097 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16194.39 MB 2025-02-14 10:54:18,097 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16713.68 MB 2025-02-14 10:54:18,097 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 519.30 MB 2025-02-14 10:54:18,097 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17995.66 MB 2025-02-14 10:54:18,097 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22907.19 MB 2025-02-14 10:54:18,097 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4911.53 MB 2025-02-14 10:54:18,097 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20503.10 MB 2025-02-14 10:54:19,561 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 10:54:19,561 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 10:54:19,561 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.46 seconds 2025-02-14 10:54:19,561 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:54:19,561 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16713.68 MB 2025-02-14 10:54:19,561 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17119.78 MB 2025-02-14 10:54:19,561 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 406.09 MB 2025-02-14 10:54:19,561 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22907.19 MB 2025-02-14 10:54:19,562 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19061.01 MB 2025-02-14 10:54:19,562 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3846.18 MB 2025-02-14 10:54:19,562 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21055.76 MB 2025-02-14 10:54:19,573 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 10:54:19,573 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 10:54:19,573 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:54:19,573 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:54:19,573 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17119.78 MB 2025-02-14 10:54:19,573 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18564.98 MB 2025-02-14 10:54:19,573 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1445.20 MB 2025-02-14 10:54:19,573 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19061.01 MB 2025-02-14 10:54:19,573 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21231.57 MB 2025-02-14 10:54:19,573 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2170.55 MB 2025-02-14 10:54:19,573 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19649.31 MB 2025-02-14 10:54:19,738 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 10:54:19,738 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 10:54:19,738 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 10:54:19,738 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:54:19,738 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18564.98 MB 2025-02-14 10:54:19,738 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20280.01 MB 2025-02-14 10:54:19,738 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1715.04 MB 2025-02-14 10:54:19,738 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21231.57 MB 2025-02-14 10:54:19,738 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26298.29 MB 2025-02-14 10:54:19,738 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5066.72 MB 2025-02-14 10:54:19,738 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24521.37 MB 2025-02-14 10:54:19,738 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 10:54:19,738 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 10:54:19,738 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-14 10:54:19,738 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:54:19,738 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17119.78 MB 2025-02-14 10:54:19,738 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20280.01 MB 2025-02-14 10:54:19,738 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3160.24 MB 2025-02-14 10:54:19,738 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19061.01 MB 2025-02-14 10:54:19,738 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26298.29 MB 2025-02-14 10:54:19,738 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7237.27 MB 2025-02-14 10:54:19,739 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24521.37 MB 2025-02-14 10:54:19,868 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 10:54:19,868 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 10:54:19,868 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 10:54:19,868 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:54:19,868 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21453.17 MB 2025-02-14 10:54:19,868 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22039.93 MB 2025-02-14 10:54:19,868 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 586.76 MB 2025-02-14 10:54:19,868 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26298.29 MB 2025-02-14 10:54:19,868 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26617.05 MB 2025-02-14 10:54:19,868 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 318.77 MB 2025-02-14 10:54:19,868 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22581.39 MB 2025-02-14 10:54:19,883 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 10:54:19,883 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 10:54:19,883 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:54:19,883 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:54:19,883 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22355.79 MB 2025-02-14 10:54:19,883 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22562.22 MB 2025-02-14 10:54:19,883 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.43 MB 2025-02-14 10:54:19,883 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26617.05 MB 2025-02-14 10:54:19,883 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26621.25 MB 2025-02-14 10:54:19,884 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-14 10:54:19,884 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22703.06 MB 2025-02-14 10:54:19,885 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 10:54:19,885 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 10:54:19,885 - resource_logging.py:150 - __exit__ - DEBUG - Time: 6.52 seconds 2025-02-14 10:54:19,885 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:54:19,885 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14038.32 MB 2025-02-14 10:54:19,885 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22763.29 MB 2025-02-14 10:54:19,885 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8724.98 MB 2025-02-14 10:54:19,885 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52661.58 MB 2025-02-14 10:54:19,885 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26621.25 MB 2025-02-14 10:54:19,885 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26040.34 MB 2025-02-14 10:54:19,885 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22763.29 MB 2025-02-14 10:54:20,152 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 10:54:20,152 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 10:54:20,152 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 10:54:20,152 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:54:20,152 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22763.29 MB 2025-02-14 10:54:20,152 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25777.33 MB 2025-02-14 10:54:20,152 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-14 10:54:20,152 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26621.25 MB 2025-02-14 10:54:20,152 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27023.90 MB 2025-02-14 10:54:20,152 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 402.65 MB 2025-02-14 10:54:20,152 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26078.96 MB 2025-02-14 10:54:20,179 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 10:54:20,179 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 10:54:20,186 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 10:54:20,186 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 10:54:20,186 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 10:54:20,186 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:54:20,187 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18598.57 MB 2025-02-14 10:54:20,187 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27037.59 MB 2025-02-14 10:54:20,187 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 10:54:20,187 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27023.90 MB 2025-02-14 10:54:20,187 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37513.85 MB 2025-02-14 10:54:20,187 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 10:54:20,187 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27037.59 MB 2025-02-14 10:54:20,350 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 10:54:20,351 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:54:20,351 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 10:54:20,352 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:54:20,352 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 10:54:20,357 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 10:54:20,358 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:54:20,358 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 10:54:20,358 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 10:55:31,062 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:55:31,062 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 10:55:31,067 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 10:55:31,071 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:55:31,071 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1692, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 10:55:31,072 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:55:31,072 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1692, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 10:55:57,009 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 10:55:57,010 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 10:55:57,010 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.93 seconds 2025-02-14 10:55:57,010 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:55:57,010 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24758.83 MB 2025-02-14 10:55:57,010 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30746.73 MB 2025-02-14 10:55:57,010 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5987.89 MB 2025-02-14 10:55:57,010 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50098.86 MB 2025-02-14 10:55:57,010 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39676.02 MB 2025-02-14 10:55:57,010 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10422.85 MB 2025-02-14 10:55:57,010 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39666.83 MB 2025-02-14 10:55:57,112 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 10:55:57,112 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 10:55:57,112 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 10:55:57,112 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:55:57,112 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30746.73 MB 2025-02-14 10:55:57,112 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24574.03 MB 2025-02-14 10:55:57,112 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6172.70 MB 2025-02-14 10:55:57,112 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39676.02 MB 2025-02-14 10:55:57,112 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57122.23 MB 2025-02-14 10:55:57,112 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 17446.21 MB 2025-02-14 10:55:57,112 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48260.99 MB 2025-02-14 10:55:59,023 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 10:55:59,023 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 10:55:59,023 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 10:55:59,023 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:55:59,023 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24574.03 MB 2025-02-14 10:55:59,023 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25104.87 MB 2025-02-14 10:55:59,023 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 10:55:59,023 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57122.23 MB 2025-02-14 10:55:59,023 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30907.83 MB 2025-02-14 10:55:59,023 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26214.40 MB 2025-02-14 10:55:59,023 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29084.21 MB 2025-02-14 10:55:59,036 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 10:55:59,037 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 10:55:59,037 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:55:59,037 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:55:59,037 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25104.87 MB 2025-02-14 10:55:59,037 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26994.41 MB 2025-02-14 10:55:59,037 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 10:55:59,037 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30907.83 MB 2025-02-14 10:55:59,037 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30907.83 MB 2025-02-14 10:55:59,037 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:55:59,037 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28411.84 MB 2025-02-14 10:55:59,252 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 10:55:59,252 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 10:55:59,252 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 10:55:59,252 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:55:59,252 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26994.41 MB 2025-02-14 10:55:59,252 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29236.26 MB 2025-02-14 10:55:59,252 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 10:55:59,252 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30907.83 MB 2025-02-14 10:55:59,252 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37513.85 MB 2025-02-14 10:55:59,252 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 10:55:59,252 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34780.54 MB 2025-02-14 10:55:59,253 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 10:55:59,253 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 10:55:59,253 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 10:55:59,253 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:55:59,253 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25104.87 MB 2025-02-14 10:55:59,253 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29236.26 MB 2025-02-14 10:55:59,253 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 10:55:59,253 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30907.83 MB 2025-02-14 10:55:59,253 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37513.85 MB 2025-02-14 10:55:59,253 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 10:55:59,253 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34780.54 MB 2025-02-14 10:55:59,429 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 10:55:59,429 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 10:55:59,429 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 10:55:59,429 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:55:59,429 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30769.80 MB 2025-02-14 10:55:59,429 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31536.81 MB 2025-02-14 10:55:59,429 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 10:55:59,429 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37513.85 MB 2025-02-14 10:55:59,429 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37931.19 MB 2025-02-14 10:55:59,429 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 10:55:59,429 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32244.60 MB 2025-02-14 10:55:59,448 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 10:55:59,448 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 10:55:59,448 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:55:59,448 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:55:59,448 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31949.70 MB 2025-02-14 10:55:59,448 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32178.28 MB 2025-02-14 10:55:59,448 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.58 MB 2025-02-14 10:55:59,448 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37931.19 MB 2025-02-14 10:55:59,448 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37931.19 MB 2025-02-14 10:55:59,448 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:55:59,448 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32402.59 MB 2025-02-14 10:55:59,449 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 10:55:59,450 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 10:55:59,450 - resource_logging.py:150 - __exit__ - DEBUG - Time: 28.38 seconds 2025-02-14 10:55:59,450 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:55:59,450 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18863.77 MB 2025-02-14 10:55:59,450 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32379.05 MB 2025-02-14 10:55:59,450 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13515.28 MB 2025-02-14 10:55:59,450 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50098.86 MB 2025-02-14 10:55:59,450 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37931.19 MB 2025-02-14 10:55:59,450 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12167.68 MB 2025-02-14 10:55:59,450 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32402.59 MB 2025-02-14 10:55:59,717 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 10:55:59,717 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 10:55:59,717 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 10:55:59,717 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:55:59,717 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32379.05 MB 2025-02-14 10:55:59,717 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23863.59 MB 2025-02-14 10:55:59,717 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8515.47 MB 2025-02-14 10:55:59,717 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37931.19 MB 2025-02-14 10:55:59,717 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37931.19 MB 2025-02-14 10:55:59,717 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:55:59,717 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34887.03 MB 2025-02-14 10:55:59,735 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8150, cut from 8152 2025-02-14 10:55:59,735 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 10:55:59,741 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 10:55:59,741 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 10:55:59,741 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:55:59,741 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:55:59,741 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23863.59 MB 2025-02-14 10:55:59,741 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32290.09 MB 2025-02-14 10:55:59,741 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8426.50 MB 2025-02-14 10:55:59,741 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37931.19 MB 2025-02-14 10:55:59,741 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46309.31 MB 2025-02-14 10:55:59,741 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8378.12 MB 2025-02-14 10:55:59,741 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32290.09 MB 2025-02-14 10:55:59,913 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7942] 2025-02-14 10:55:59,915 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:55:59,915 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 10:55:59,916 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:55:59,916 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 10:55:59,921 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 10:55:59,922 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:55:59,922 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 10:55:59,922 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 10:56:47,501 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:56:47,501 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 10:56:47,506 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 10:56:47,510 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:56:47,510 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1589, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 10:56:47,511 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:56:47,511 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1589, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 10:57:11,990 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 10:57:11,990 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 10:57:11,990 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.47 seconds 2025-02-14 10:57:11,990 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:57:11,990 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24041.11 MB 2025-02-14 10:57:11,990 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29664.49 MB 2025-02-14 10:57:11,990 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5623.38 MB 2025-02-14 10:57:11,990 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58875.45 MB 2025-02-14 10:57:11,990 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39288.05 MB 2025-02-14 10:57:11,990 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19587.40 MB 2025-02-14 10:57:11,990 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38496.12 MB 2025-02-14 10:57:12,085 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 10:57:12,085 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 10:57:12,085 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 10:57:12,085 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:57:12,085 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29664.49 MB 2025-02-14 10:57:12,085 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24038.57 MB 2025-02-14 10:57:12,085 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5625.93 MB 2025-02-14 10:57:12,085 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39288.05 MB 2025-02-14 10:57:12,085 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51269.07 MB 2025-02-14 10:57:12,085 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 11981.03 MB 2025-02-14 10:57:12,085 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45790.75 MB 2025-02-14 10:57:13,997 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 10:57:13,997 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 10:57:13,997 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 10:57:13,997 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:57:13,997 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24038.57 MB 2025-02-14 10:57:13,998 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24569.41 MB 2025-02-14 10:57:13,998 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 10:57:13,998 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51269.07 MB 2025-02-14 10:57:13,998 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29475.47 MB 2025-02-14 10:57:13,998 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21793.60 MB 2025-02-14 10:57:13,998 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28548.74 MB 2025-02-14 10:57:14,013 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 10:57:14,013 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 10:57:14,013 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:57:14,013 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:57:14,013 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24569.41 MB 2025-02-14 10:57:14,013 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26458.94 MB 2025-02-14 10:57:14,013 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 10:57:14,013 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29475.47 MB 2025-02-14 10:57:14,013 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30419.19 MB 2025-02-14 10:57:14,013 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 10:57:14,013 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27876.37 MB 2025-02-14 10:57:14,237 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 10:57:14,237 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 10:57:14,237 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 10:57:14,237 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:57:14,237 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26458.94 MB 2025-02-14 10:57:14,237 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28700.80 MB 2025-02-14 10:57:14,237 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 10:57:14,237 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30419.19 MB 2025-02-14 10:57:14,237 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36553.36 MB 2025-02-14 10:57:14,237 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 10:57:14,237 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34245.08 MB 2025-02-14 10:57:14,238 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 10:57:14,238 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 10:57:14,238 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.24 seconds 2025-02-14 10:57:14,238 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:57:14,238 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24569.41 MB 2025-02-14 10:57:14,238 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28700.80 MB 2025-02-14 10:57:14,238 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 10:57:14,238 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29475.47 MB 2025-02-14 10:57:14,238 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36553.36 MB 2025-02-14 10:57:14,238 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7077.89 MB 2025-02-14 10:57:14,238 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34245.08 MB 2025-02-14 10:57:14,411 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 10:57:14,411 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 10:57:14,411 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 10:57:14,411 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:57:14,411 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30234.34 MB 2025-02-14 10:57:14,411 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31001.34 MB 2025-02-14 10:57:14,411 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 10:57:14,411 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36553.36 MB 2025-02-14 10:57:14,411 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36968.60 MB 2025-02-14 10:57:14,411 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 10:57:14,411 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31709.13 MB 2025-02-14 10:57:14,430 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 10:57:14,430 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 10:57:14,430 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:57:14,430 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:57:14,430 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31414.23 MB 2025-02-14 10:57:14,430 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31642.31 MB 2025-02-14 10:57:14,430 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.08 MB 2025-02-14 10:57:14,430 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36968.60 MB 2025-02-14 10:57:14,430 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36968.60 MB 2025-02-14 10:57:14,430 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:57:14,430 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31843.47 MB 2025-02-14 10:57:14,431 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 10:57:14,431 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 10:57:14,431 - resource_logging.py:150 - __exit__ - DEBUG - Time: 26.92 seconds 2025-02-14 10:57:14,431 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:57:14,431 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18504.91 MB 2025-02-14 10:57:14,431 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31842.67 MB 2025-02-14 10:57:14,431 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13337.76 MB 2025-02-14 10:57:14,431 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58875.45 MB 2025-02-14 10:57:14,431 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36968.60 MB 2025-02-14 10:57:14,431 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21906.85 MB 2025-02-14 10:57:14,431 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31843.47 MB 2025-02-14 10:57:14,698 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 10:57:14,698 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 10:57:14,698 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 10:57:14,698 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:57:14,698 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31842.67 MB 2025-02-14 10:57:14,698 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23498.25 MB 2025-02-14 10:57:14,698 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8344.41 MB 2025-02-14 10:57:14,698 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36968.60 MB 2025-02-14 10:57:14,698 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36968.60 MB 2025-02-14 10:57:14,698 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:57:14,698 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34345.42 MB 2025-02-14 10:57:14,716 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8133, cut from 8135 2025-02-14 10:57:14,716 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 10:57:14,722 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 10:57:14,722 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 10:57:14,722 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:57:14,722 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:57:14,722 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23498.25 MB 2025-02-14 10:57:14,722 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31907.03 MB 2025-02-14 10:57:14,722 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8408.77 MB 2025-02-14 10:57:14,722 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36968.60 MB 2025-02-14 10:57:14,722 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41148.22 MB 2025-02-14 10:57:14,722 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4179.62 MB 2025-02-14 10:57:14,722 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31907.03 MB 2025-02-14 10:57:14,884 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7925] 2025-02-14 10:57:14,885 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:57:14,885 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 10:57:14,886 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:57:14,886 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 10:57:14,891 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 10:57:14,892 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:57:14,892 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 10:57:14,892 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 10:58:11,852 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:58:11,852 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 10:58:11,857 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 10:58:11,861 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:58:11,861 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1285, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 10:58:11,862 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:58:11,862 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1285, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 10:58:31,648 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 10:58:31,648 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 10:58:31,648 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.78 seconds 2025-02-14 10:58:31,648 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:58:31,648 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21922.79 MB 2025-02-14 10:58:31,648 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26470.34 MB 2025-02-14 10:58:31,648 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4547.54 MB 2025-02-14 10:58:31,648 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49507.47 MB 2025-02-14 10:58:31,648 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38184.94 MB 2025-02-14 10:58:31,648 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11322.52 MB 2025-02-14 10:58:31,648 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35471.93 MB 2025-02-14 10:58:31,722 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 10:58:31,722 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 10:58:31,722 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 10:58:31,723 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:58:31,723 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26470.34 MB 2025-02-14 10:58:31,723 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22458.17 MB 2025-02-14 10:58:31,723 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4012.17 MB 2025-02-14 10:58:31,723 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38184.94 MB 2025-02-14 10:58:31,723 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47183.82 MB 2025-02-14 10:58:31,723 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8998.88 MB 2025-02-14 10:58:31,723 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40019.81 MB 2025-02-14 10:58:33,625 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 10:58:33,625 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 10:58:33,625 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.90 seconds 2025-02-14 10:58:33,625 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:58:33,625 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22458.17 MB 2025-02-14 10:58:33,625 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22989.01 MB 2025-02-14 10:58:33,625 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 10:58:33,625 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47183.82 MB 2025-02-14 10:58:33,625 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33636.22 MB 2025-02-14 10:58:33,625 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13547.60 MB 2025-02-14 10:58:33,625 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26968.34 MB 2025-02-14 10:58:33,638 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 10:58:33,639 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 10:58:33,639 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:58:33,639 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:58:33,639 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22989.01 MB 2025-02-14 10:58:33,639 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24878.54 MB 2025-02-14 10:58:33,639 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 10:58:33,639 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33636.22 MB 2025-02-14 10:58:33,639 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33636.22 MB 2025-02-14 10:58:33,639 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:58:33,639 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26295.97 MB 2025-02-14 10:58:33,865 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 10:58:33,865 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 10:58:33,865 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 10:58:33,865 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:58:33,865 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24878.54 MB 2025-02-14 10:58:33,865 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27120.40 MB 2025-02-14 10:58:33,865 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 10:58:33,865 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33636.22 MB 2025-02-14 10:58:33,865 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35523.66 MB 2025-02-14 10:58:33,865 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 10:58:33,865 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32664.68 MB 2025-02-14 10:58:33,866 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 10:58:33,866 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 10:58:33,866 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.24 seconds 2025-02-14 10:58:33,866 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:58:33,866 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22989.01 MB 2025-02-14 10:58:33,866 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27120.40 MB 2025-02-14 10:58:33,866 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 10:58:33,866 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33636.22 MB 2025-02-14 10:58:33,866 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35523.66 MB 2025-02-14 10:58:33,866 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 10:58:33,866 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32664.68 MB 2025-02-14 10:58:34,035 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 10:58:34,035 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 10:58:34,035 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 10:58:34,035 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:58:34,035 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28653.94 MB 2025-02-14 10:58:34,035 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29420.94 MB 2025-02-14 10:58:34,035 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 10:58:34,035 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35523.66 MB 2025-02-14 10:58:34,035 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35938.89 MB 2025-02-14 10:58:34,035 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 10:58:34,035 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30128.73 MB 2025-02-14 10:58:34,054 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 10:58:34,054 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 10:58:34,054 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:58:34,054 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:58:34,054 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29833.83 MB 2025-02-14 10:58:34,054 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30062.83 MB 2025-02-14 10:58:34,054 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.00 MB 2025-02-14 10:58:34,054 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35938.89 MB 2025-02-14 10:58:34,054 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35938.89 MB 2025-02-14 10:58:34,054 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:58:34,054 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30303.90 MB 2025-02-14 10:58:34,055 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 10:58:34,055 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 10:58:34,055 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.19 seconds 2025-02-14 10:58:34,055 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:58:34,055 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17445.75 MB 2025-02-14 10:58:34,055 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30263.31 MB 2025-02-14 10:58:34,055 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12817.56 MB 2025-02-14 10:58:34,055 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49507.47 MB 2025-02-14 10:58:34,056 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35938.89 MB 2025-02-14 10:58:34,056 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13568.57 MB 2025-02-14 10:58:34,056 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30303.90 MB 2025-02-14 10:58:34,323 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 10:58:34,323 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 10:58:34,323 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 10:58:34,323 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:58:34,323 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30263.31 MB 2025-02-14 10:58:34,323 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22441.00 MB 2025-02-14 10:58:34,323 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7822.31 MB 2025-02-14 10:58:34,323 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35938.89 MB 2025-02-14 10:58:34,323 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35938.89 MB 2025-02-14 10:58:34,323 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:58:34,323 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32767.60 MB 2025-02-14 10:58:34,341 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8138, cut from 8140 2025-02-14 10:58:34,341 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 10:58:34,347 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 10:58:34,347 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 10:58:34,347 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:58:34,347 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:58:34,347 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22441.00 MB 2025-02-14 10:58:34,347 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30854.98 MB 2025-02-14 10:58:34,347 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8413.98 MB 2025-02-14 10:58:34,347 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35938.89 MB 2025-02-14 10:58:34,347 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44304.43 MB 2025-02-14 10:58:34,347 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8365.54 MB 2025-02-14 10:58:34,347 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30854.98 MB 2025-02-14 10:58:34,510 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7930] 2025-02-14 10:58:34,511 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:58:34,511 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 10:58:34,512 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:58:34,512 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 10:58:34,517 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 10:58:34,518 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:58:34,518 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 10:58:34,518 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 10:59:25,395 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:59:25,395 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 10:59:25,401 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 10:59:25,405 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:59:25,405 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1466, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 10:59:25,406 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:59:25,406 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1466, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 10:59:47,958 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 10:59:47,958 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 10:59:47,958 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.54 seconds 2025-02-14 10:59:47,958 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:59:47,958 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23184.03 MB 2025-02-14 10:59:47,958 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28372.38 MB 2025-02-14 10:59:47,958 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5188.35 MB 2025-02-14 10:59:47,958 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56851.69 MB 2025-02-14 10:59:47,958 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38824.57 MB 2025-02-14 10:59:47,958 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18027.12 MB 2025-02-14 10:59:47,958 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37185.25 MB 2025-02-14 10:59:48,040 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 10:59:48,040 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 10:59:48,040 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 10:59:48,040 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:59:48,040 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28372.38 MB 2025-02-14 10:59:48,040 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23399.13 MB 2025-02-14 10:59:48,040 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4973.26 MB 2025-02-14 10:59:48,040 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38824.57 MB 2025-02-14 10:59:48,040 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48188.36 MB 2025-02-14 10:59:48,040 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9363.78 MB 2025-02-14 10:59:48,040 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42488.24 MB 2025-02-14 10:59:49,959 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 10:59:49,959 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 10:59:49,959 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 10:59:49,959 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:59:49,959 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23399.13 MB 2025-02-14 10:59:49,959 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23929.97 MB 2025-02-14 10:59:49,959 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 10:59:49,959 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48188.36 MB 2025-02-14 10:59:49,959 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29456.60 MB 2025-02-14 10:59:49,959 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18731.76 MB 2025-02-14 10:59:49,959 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27909.30 MB 2025-02-14 10:59:49,974 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 10:59:49,974 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 10:59:49,974 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 10:59:49,974 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:59:49,974 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23929.97 MB 2025-02-14 10:59:49,974 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25819.50 MB 2025-02-14 10:59:49,974 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 10:59:49,974 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29456.60 MB 2025-02-14 10:59:49,974 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30400.32 MB 2025-02-14 10:59:49,974 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 10:59:49,974 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27236.93 MB 2025-02-14 10:59:50,188 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 10:59:50,188 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 10:59:50,188 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 10:59:50,188 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:59:50,188 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25819.50 MB 2025-02-14 10:59:50,188 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28061.36 MB 2025-02-14 10:59:50,188 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 10:59:50,188 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30400.32 MB 2025-02-14 10:59:50,188 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36062.63 MB 2025-02-14 10:59:50,188 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 10:59:50,188 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33605.64 MB 2025-02-14 10:59:50,188 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 10:59:50,188 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 10:59:50,188 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 10:59:50,188 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:59:50,188 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23929.97 MB 2025-02-14 10:59:50,188 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28061.36 MB 2025-02-14 10:59:50,188 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 10:59:50,188 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29456.60 MB 2025-02-14 10:59:50,188 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36062.63 MB 2025-02-14 10:59:50,189 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 10:59:50,189 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33605.64 MB 2025-02-14 10:59:50,360 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 10:59:50,360 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 10:59:50,360 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 10:59:50,360 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:59:50,360 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29594.90 MB 2025-02-14 10:59:50,360 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30361.90 MB 2025-02-14 10:59:50,360 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 10:59:50,360 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36062.63 MB 2025-02-14 10:59:50,360 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36477.86 MB 2025-02-14 10:59:50,360 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 10:59:50,360 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31069.69 MB 2025-02-14 10:59:50,379 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 10:59:50,379 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 10:59:50,379 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:59:50,379 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:59:50,379 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30774.79 MB 2025-02-14 10:59:50,379 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31003.08 MB 2025-02-14 10:59:50,379 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.28 MB 2025-02-14 10:59:50,379 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36477.86 MB 2025-02-14 10:59:50,379 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36477.86 MB 2025-02-14 10:59:50,379 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:59:50,379 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31248.45 MB 2025-02-14 10:59:50,380 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 10:59:50,380 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 10:59:50,381 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.97 seconds 2025-02-14 10:59:50,381 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:59:50,381 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18076.37 MB 2025-02-14 10:59:50,381 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31203.19 MB 2025-02-14 10:59:50,381 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13126.82 MB 2025-02-14 10:59:50,381 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56851.69 MB 2025-02-14 10:59:50,381 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36477.86 MB 2025-02-14 10:59:50,381 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20373.83 MB 2025-02-14 10:59:50,381 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31248.45 MB 2025-02-14 10:59:50,649 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 10:59:50,649 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 10:59:50,650 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 10:59:50,650 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:59:50,650 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31203.19 MB 2025-02-14 10:59:50,650 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23065.90 MB 2025-02-14 10:59:50,650 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8137.29 MB 2025-02-14 10:59:50,650 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36477.86 MB 2025-02-14 10:59:50,650 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36477.86 MB 2025-02-14 10:59:50,650 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 10:59:50,650 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33702.88 MB 2025-02-14 10:59:50,667 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8123, cut from 8125 2025-02-14 10:59:50,668 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 10:59:50,674 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 10:59:50,674 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 10:59:50,674 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 10:59:50,674 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 10:59:50,674 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23065.90 MB 2025-02-14 10:59:50,674 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31465.75 MB 2025-02-14 10:59:50,674 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8399.85 MB 2025-02-14 10:59:50,674 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36477.86 MB 2025-02-14 10:59:50,674 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44828.72 MB 2025-02-14 10:59:50,674 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8350.86 MB 2025-02-14 10:59:50,674 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31465.75 MB 2025-02-14 10:59:50,839 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7915] 2025-02-14 10:59:50,840 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:59:50,840 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 10:59:50,841 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:59:50,841 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 10:59:50,846 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 10:59:50,847 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:59:50,847 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 10:59:50,847 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 10:59:59,300 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:59:59,300 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 10:59:59,305 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 10:59:59,309 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:59:59,309 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1126, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 10:59:59,310 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 10:59:59,310 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1126, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 11:00:16,907 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 11:00:16,907 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 11:00:16,908 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.59 seconds 2025-02-14 11:00:16,908 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:00:16,908 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20814.86 MB 2025-02-14 11:00:16,908 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24799.71 MB 2025-02-14 11:00:16,908 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3984.85 MB 2025-02-14 11:00:16,908 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53179.58 MB 2025-02-14 11:00:16,908 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29253.17 MB 2025-02-14 11:00:16,908 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23926.41 MB 2025-02-14 11:00:16,908 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33684.42 MB 2025-02-14 11:00:16,995 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 11:00:16,995 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 11:00:16,995 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 11:00:16,995 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:00:16,995 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24799.71 MB 2025-02-14 11:00:16,995 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21631.57 MB 2025-02-14 11:00:16,995 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3168.13 MB 2025-02-14 11:00:16,995 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29253.17 MB 2025-02-14 11:00:16,995 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42461.04 MB 2025-02-14 11:00:16,995 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 13207.86 MB 2025-02-14 11:00:16,995 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36337.52 MB 2025-02-14 11:00:18,940 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 11:00:18,940 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 11:00:18,940 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-14 11:00:18,940 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:00:18,940 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21631.57 MB 2025-02-14 11:00:18,940 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22162.42 MB 2025-02-14 11:00:18,940 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 11:00:18,940 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42461.04 MB 2025-02-14 11:00:18,940 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26682.06 MB 2025-02-14 11:00:18,940 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15778.97 MB 2025-02-14 11:00:18,940 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26141.75 MB 2025-02-14 11:00:18,954 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 11:00:18,954 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 11:00:18,954 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:00:18,954 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:00:18,954 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22162.42 MB 2025-02-14 11:00:18,954 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24051.95 MB 2025-02-14 11:00:18,954 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 11:00:18,954 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26682.06 MB 2025-02-14 11:00:18,954 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28569.50 MB 2025-02-14 11:00:18,954 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 11:00:18,954 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25469.38 MB 2025-02-14 11:00:19,167 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 11:00:19,167 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 11:00:19,167 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 11:00:19,167 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:00:19,167 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24051.95 MB 2025-02-14 11:00:19,167 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26293.81 MB 2025-02-14 11:00:19,167 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 11:00:19,167 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28569.50 MB 2025-02-14 11:00:19,167 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34231.81 MB 2025-02-14 11:00:19,167 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 11:00:19,167 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31838.09 MB 2025-02-14 11:00:19,168 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 11:00:19,168 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 11:00:19,168 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 11:00:19,168 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:00:19,168 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22162.42 MB 2025-02-14 11:00:19,168 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26293.81 MB 2025-02-14 11:00:19,168 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 11:00:19,168 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26682.06 MB 2025-02-14 11:00:19,168 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34231.81 MB 2025-02-14 11:00:19,168 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-14 11:00:19,168 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31838.09 MB 2025-02-14 11:00:19,337 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 11:00:19,337 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 11:00:19,337 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 11:00:19,337 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:00:19,337 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27827.35 MB 2025-02-14 11:00:19,337 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28594.35 MB 2025-02-14 11:00:19,337 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 11:00:19,337 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34231.81 MB 2025-02-14 11:00:19,337 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34644.95 MB 2025-02-14 11:00:19,337 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 11:00:19,337 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29302.14 MB 2025-02-14 11:00:19,356 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 11:00:19,356 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 11:00:19,356 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:00:19,356 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:00:19,356 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29007.24 MB 2025-02-14 11:00:19,356 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29235.08 MB 2025-02-14 11:00:19,356 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.84 MB 2025-02-14 11:00:19,356 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34644.95 MB 2025-02-14 11:00:19,356 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34644.95 MB 2025-02-14 11:00:19,356 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:00:19,356 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29433.30 MB 2025-02-14 11:00:19,358 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 11:00:19,358 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 11:00:19,358 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.05 seconds 2025-02-14 11:00:19,358 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:00:19,358 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16891.78 MB 2025-02-14 11:00:19,358 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29435.47 MB 2025-02-14 11:00:19,358 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12543.69 MB 2025-02-14 11:00:19,358 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53179.58 MB 2025-02-14 11:00:19,358 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34644.95 MB 2025-02-14 11:00:19,358 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18534.63 MB 2025-02-14 11:00:19,358 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29435.47 MB 2025-02-14 11:00:19,629 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 11:00:19,629 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 11:00:19,629 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 11:00:19,629 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:00:19,629 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29435.47 MB 2025-02-14 11:00:19,629 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21885.50 MB 2025-02-14 11:00:19,629 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7549.96 MB 2025-02-14 11:00:19,629 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34644.95 MB 2025-02-14 11:00:19,629 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34644.95 MB 2025-02-14 11:00:19,629 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:00:19,629 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31938.53 MB 2025-02-14 11:00:19,647 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8134, cut from 8136 2025-02-14 11:00:19,647 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 11:00:19,653 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 11:00:19,653 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 11:00:19,653 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:00:19,653 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:00:19,653 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21885.50 MB 2025-02-14 11:00:19,654 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30295.31 MB 2025-02-14 11:00:19,654 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8409.81 MB 2025-02-14 11:00:19,654 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34644.95 MB 2025-02-14 11:00:19,654 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43006.30 MB 2025-02-14 11:00:19,654 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8361.35 MB 2025-02-14 11:00:19,654 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30295.31 MB 2025-02-14 11:00:19,815 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7926] 2025-02-14 11:00:19,816 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:00:19,816 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 11:00:19,817 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:00:19,817 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 11:00:19,822 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 11:00:19,823 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:00:19,823 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 11:00:19,823 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 11:01:52,151 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:01:52,152 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 11:01:52,157 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 11:01:52,161 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:01:52,161 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 153, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 11:01:52,162 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:01:52,162 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 153, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 11:01:54,542 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 11:01:54,542 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 11:01:54,542 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.38 seconds 2025-02-14 11:01:54,542 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:01:54,542 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14034.84 MB 2025-02-14 11:01:54,542 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14576.29 MB 2025-02-14 11:01:54,542 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 541.46 MB 2025-02-14 11:01:54,542 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55547.27 MB 2025-02-14 11:01:54,542 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17855.15 MB 2025-02-14 11:01:54,542 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -37692.11 MB 2025-02-14 11:01:54,542 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23507.01 MB 2025-02-14 11:01:54,554 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 11:01:54,554 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 11:01:54,554 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:01:54,554 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:01:54,554 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14576.29 MB 2025-02-14 11:01:54,554 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14796.49 MB 2025-02-14 11:01:54,554 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 220.20 MB 2025-02-14 11:01:54,554 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17855.15 MB 2025-02-14 11:01:54,554 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18377.34 MB 2025-02-14 11:01:54,554 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 522.19 MB 2025-02-14 11:01:54,554 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16672.98 MB 2025-02-14 11:01:55,259 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 11:01:55,259 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 11:01:55,259 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.70 seconds 2025-02-14 11:01:55,259 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:01:55,259 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14796.49 MB 2025-02-14 11:01:55,259 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14991.58 MB 2025-02-14 11:01:55,259 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 195.08 MB 2025-02-14 11:01:55,259 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18377.34 MB 2025-02-14 11:01:55,259 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17905.48 MB 2025-02-14 11:01:55,259 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -471.86 MB 2025-02-14 11:01:55,259 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18967.97 MB 2025-02-14 11:01:55,266 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 11:01:55,266 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 11:01:55,266 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 11:01:55,266 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:01:55,266 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14991.51 MB 2025-02-14 11:01:55,266 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15685.75 MB 2025-02-14 11:01:55,266 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 694.24 MB 2025-02-14 11:01:55,266 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17905.48 MB 2025-02-14 11:01:55,266 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17905.48 MB 2025-02-14 11:01:55,266 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:01:55,266 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16206.66 MB 2025-02-14 11:01:55,347 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 11:01:55,347 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 11:01:55,347 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 11:01:55,347 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:01:55,347 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15685.75 MB 2025-02-14 11:01:55,347 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16509.67 MB 2025-02-14 11:01:55,347 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 823.92 MB 2025-02-14 11:01:55,347 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17905.48 MB 2025-02-14 11:01:55,347 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19297.99 MB 2025-02-14 11:01:55,347 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1392.51 MB 2025-02-14 11:01:55,347 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18547.15 MB 2025-02-14 11:01:55,347 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 11:01:55,347 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 11:01:55,348 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 11:01:55,348 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:01:55,348 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14991.51 MB 2025-02-14 11:01:55,348 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16509.67 MB 2025-02-14 11:01:55,348 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1518.16 MB 2025-02-14 11:01:55,348 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17905.48 MB 2025-02-14 11:01:55,348 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19297.99 MB 2025-02-14 11:01:55,348 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1392.51 MB 2025-02-14 11:01:55,348 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18547.15 MB 2025-02-14 11:01:55,411 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 11:01:55,411 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 11:01:55,411 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 11:01:55,411 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:01:55,411 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17073.25 MB 2025-02-14 11:01:55,411 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17355.12 MB 2025-02-14 11:01:55,411 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 281.87 MB 2025-02-14 11:01:55,411 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19297.99 MB 2025-02-14 11:01:55,411 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19444.79 MB 2025-02-14 11:01:55,411 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 146.80 MB 2025-02-14 11:01:55,411 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17626.13 MB 2025-02-14 11:01:55,421 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 11:01:55,421 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 11:01:55,421 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:01:55,421 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:01:55,421 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17506.86 MB 2025-02-14 11:01:55,421 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17714.94 MB 2025-02-14 11:01:55,421 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 208.08 MB 2025-02-14 11:01:55,421 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19444.79 MB 2025-02-14 11:01:55,421 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19446.89 MB 2025-02-14 11:01:55,421 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-14 11:01:55,421 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17718.72 MB 2025-02-14 11:01:55,422 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 11:01:55,422 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 11:01:55,422 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.26 seconds 2025-02-14 11:01:55,422 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:01:55,422 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13501.77 MB 2025-02-14 11:01:55,422 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17915.67 MB 2025-02-14 11:01:55,422 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4413.90 MB 2025-02-14 11:01:55,422 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55547.27 MB 2025-02-14 11:01:55,422 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19446.89 MB 2025-02-14 11:01:55,422 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -36100.37 MB 2025-02-14 11:01:55,422 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17915.67 MB 2025-02-14 11:01:55,689 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 11:01:55,689 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 11:01:55,689 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 11:01:55,689 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:01:55,689 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17915.67 MB 2025-02-14 11:01:55,689 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17306.71 MB 2025-02-14 11:01:55,689 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -608.95 MB 2025-02-14 11:01:55,689 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19446.89 MB 2025-02-14 11:01:55,689 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19715.33 MB 2025-02-14 11:01:55,689 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 268.44 MB 2025-02-14 11:01:55,689 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19019.40 MB 2025-02-14 11:01:55,706 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8148, cut from 8150 2025-02-14 11:01:55,707 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 11:01:55,713 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 11:01:55,713 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 11:01:55,713 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:01:55,713 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:01:55,713 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17306.71 MB 2025-02-14 11:01:55,713 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25731.66 MB 2025-02-14 11:01:55,713 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8424.95 MB 2025-02-14 11:01:55,713 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19715.33 MB 2025-02-14 11:01:55,713 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30186.41 MB 2025-02-14 11:01:55,713 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10471.08 MB 2025-02-14 11:01:55,713 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25731.66 MB 2025-02-14 11:01:55,877 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7940] 2025-02-14 11:01:55,878 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:01:55,878 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 11:01:55,879 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:01:55,879 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 11:01:55,884 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 11:01:55,885 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:01:55,885 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 11:01:55,885 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 11:02:48,543 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:02:48,543 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 11:02:48,548 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 11:02:48,552 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:02:48,552 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1941, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 11:02:48,553 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:02:48,553 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1941, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 11:03:18,299 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 11:03:18,299 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 11:03:18,299 - resource_logging.py:150 - __exit__ - DEBUG - Time: 29.74 seconds 2025-02-14 11:03:18,299 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:03:18,299 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26493.90 MB 2025-02-14 11:03:18,299 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33363.00 MB 2025-02-14 11:03:18,299 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6869.09 MB 2025-02-14 11:03:18,299 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38562.43 MB 2025-02-14 11:03:18,299 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40531.66 MB 2025-02-14 11:03:18,299 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1969.23 MB 2025-02-14 11:03:18,300 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42307.87 MB 2025-02-14 11:03:18,417 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 11:03:18,417 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 11:03:18,417 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 11:03:18,417 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:03:18,417 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33363.00 MB 2025-02-14 11:03:18,417 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25868.50 MB 2025-02-14 11:03:18,417 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7494.49 MB 2025-02-14 11:03:18,417 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40531.66 MB 2025-02-14 11:03:18,417 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 62077.80 MB 2025-02-14 11:03:18,417 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 21546.14 MB 2025-02-14 11:03:18,417 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52546.85 MB 2025-02-14 11:03:20,344 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 11:03:20,344 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 11:03:20,344 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 11:03:20,344 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:03:20,344 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25868.50 MB 2025-02-14 11:03:20,344 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26399.35 MB 2025-02-14 11:03:20,344 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 11:03:20,344 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62077.80 MB 2025-02-14 11:03:20,344 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30888.95 MB 2025-02-14 11:03:20,344 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31188.84 MB 2025-02-14 11:03:20,344 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30379.72 MB 2025-02-14 11:03:20,357 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 11:03:20,357 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 11:03:20,357 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:03:20,357 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:03:20,357 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26399.35 MB 2025-02-14 11:03:20,357 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28288.88 MB 2025-02-14 11:03:20,357 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 11:03:20,357 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30888.95 MB 2025-02-14 11:03:20,357 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32776.39 MB 2025-02-14 11:03:20,357 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 11:03:20,357 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29706.31 MB 2025-02-14 11:03:20,589 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 11:03:20,589 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 11:03:20,589 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 11:03:20,589 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:03:20,589 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28288.88 MB 2025-02-14 11:03:20,589 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30530.74 MB 2025-02-14 11:03:20,589 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 11:03:20,589 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32776.39 MB 2025-02-14 11:03:20,589 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38910.56 MB 2025-02-14 11:03:20,589 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 11:03:20,590 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36075.02 MB 2025-02-14 11:03:20,590 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 11:03:20,590 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 11:03:20,590 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.25 seconds 2025-02-14 11:03:20,590 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:03:20,590 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26399.35 MB 2025-02-14 11:03:20,590 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30530.74 MB 2025-02-14 11:03:20,590 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 11:03:20,590 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30888.95 MB 2025-02-14 11:03:20,590 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38910.56 MB 2025-02-14 11:03:20,590 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-14 11:03:20,590 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36075.02 MB 2025-02-14 11:03:20,761 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 11:03:20,761 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 11:03:20,761 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 11:03:20,761 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:03:20,761 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32064.28 MB 2025-02-14 11:03:20,761 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32831.28 MB 2025-02-14 11:03:20,761 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 11:03:20,761 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38910.56 MB 2025-02-14 11:03:20,761 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39325.79 MB 2025-02-14 11:03:20,761 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 11:03:20,761 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33539.07 MB 2025-02-14 11:03:20,780 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 11:03:20,780 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 11:03:20,780 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:03:20,780 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:03:20,780 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33244.17 MB 2025-02-14 11:03:20,780 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33473.97 MB 2025-02-14 11:03:20,780 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.80 MB 2025-02-14 11:03:20,780 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39325.79 MB 2025-02-14 11:03:20,780 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39325.79 MB 2025-02-14 11:03:20,780 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:03:20,780 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33683.96 MB 2025-02-14 11:03:20,781 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 11:03:20,781 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 11:03:20,781 - resource_logging.py:150 - __exit__ - DEBUG - Time: 32.23 seconds 2025-02-14 11:03:20,781 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:03:20,781 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19731.31 MB 2025-02-14 11:03:20,781 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33675.04 MB 2025-02-14 11:03:20,781 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13943.73 MB 2025-02-14 11:03:20,781 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38562.43 MB 2025-02-14 11:03:20,781 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39325.79 MB 2025-02-14 11:03:20,781 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 763.36 MB 2025-02-14 11:03:20,781 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33683.96 MB 2025-02-14 11:03:21,049 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 11:03:21,049 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 11:03:21,049 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 11:03:21,050 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:03:21,050 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33675.04 MB 2025-02-14 11:03:21,050 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24735.69 MB 2025-02-14 11:03:21,050 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8939.34 MB 2025-02-14 11:03:21,050 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39325.79 MB 2025-02-14 11:03:21,050 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39325.79 MB 2025-02-14 11:03:21,050 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:03:21,050 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36186.71 MB 2025-02-14 11:03:21,067 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 11:03:21,068 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 11:03:21,074 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 11:03:21,074 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 11:03:21,074 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:03:21,074 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:03:21,074 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24735.69 MB 2025-02-14 11:03:21,074 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33174.72 MB 2025-02-14 11:03:21,074 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 11:03:21,074 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39325.79 MB 2025-02-14 11:03:21,074 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47716.50 MB 2025-02-14 11:03:21,074 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 11:03:21,074 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33174.72 MB 2025-02-14 11:03:21,237 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 11:03:21,238 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:03:21,238 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 11:03:21,239 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:03:21,239 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 11:03:21,244 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 11:03:21,245 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:03:21,245 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 11:03:21,245 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 11:04:24,494 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:04:24,495 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 11:04:24,500 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 11:04:24,503 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:04:24,503 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1301, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 11:04:24,504 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:04:24,504 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1301, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 11:04:44,478 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 11:04:44,478 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 11:04:44,478 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.97 seconds 2025-02-14 11:04:44,478 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:04:44,478 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22034.28 MB 2025-02-14 11:04:44,478 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26638.45 MB 2025-02-14 11:04:44,478 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4604.17 MB 2025-02-14 11:04:44,478 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60301.51 MB 2025-02-14 11:04:44,478 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38275.12 MB 2025-02-14 11:04:44,478 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22026.39 MB 2025-02-14 11:04:44,478 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35582.52 MB 2025-02-14 11:04:44,573 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 11:04:44,573 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 11:04:44,573 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 11:04:44,573 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:04:44,573 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26638.45 MB 2025-02-14 11:04:44,573 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22541.34 MB 2025-02-14 11:04:44,573 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4097.10 MB 2025-02-14 11:04:44,573 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38275.12 MB 2025-02-14 11:04:44,573 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47330.62 MB 2025-02-14 11:04:44,573 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9055.50 MB 2025-02-14 11:04:44,573 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40271.68 MB 2025-02-14 11:04:46,491 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 11:04:46,491 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 11:04:46,491 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 11:04:46,491 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:04:46,492 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22541.34 MB 2025-02-14 11:04:46,492 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23072.19 MB 2025-02-14 11:04:46,492 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 11:04:46,492 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47330.62 MB 2025-02-14 11:04:46,492 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29481.76 MB 2025-02-14 11:04:46,492 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17848.86 MB 2025-02-14 11:04:46,492 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27051.52 MB 2025-02-14 11:04:46,505 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 11:04:46,505 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 11:04:46,505 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:04:46,505 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:04:46,505 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23072.19 MB 2025-02-14 11:04:46,505 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24961.72 MB 2025-02-14 11:04:46,505 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 11:04:46,505 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29481.76 MB 2025-02-14 11:04:46,505 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29481.76 MB 2025-02-14 11:04:46,505 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:04:46,505 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26379.15 MB 2025-02-14 11:04:46,718 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 11:04:46,718 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 11:04:46,718 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 11:04:46,718 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:04:46,718 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24961.72 MB 2025-02-14 11:04:46,718 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27203.58 MB 2025-02-14 11:04:46,718 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 11:04:46,718 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29481.76 MB 2025-02-14 11:04:46,718 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35144.07 MB 2025-02-14 11:04:46,718 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 11:04:46,719 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32747.86 MB 2025-02-14 11:04:46,719 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 11:04:46,719 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 11:04:46,719 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 11:04:46,719 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:04:46,719 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23072.19 MB 2025-02-14 11:04:46,719 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27203.58 MB 2025-02-14 11:04:46,719 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 11:04:46,719 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29481.76 MB 2025-02-14 11:04:46,719 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35144.07 MB 2025-02-14 11:04:46,719 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 11:04:46,719 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32747.86 MB 2025-02-14 11:04:46,890 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 11:04:46,890 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 11:04:46,890 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 11:04:46,890 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:04:46,890 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28737.12 MB 2025-02-14 11:04:46,890 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29504.12 MB 2025-02-14 11:04:46,890 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 11:04:46,890 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35144.07 MB 2025-02-14 11:04:46,890 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35557.21 MB 2025-02-14 11:04:46,890 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 11:04:46,890 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30211.91 MB 2025-02-14 11:04:46,909 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 11:04:46,909 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 11:04:46,909 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:04:46,909 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:04:46,909 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29917.01 MB 2025-02-14 11:04:46,909 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30143.83 MB 2025-02-14 11:04:46,909 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.82 MB 2025-02-14 11:04:46,909 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35557.21 MB 2025-02-14 11:04:46,909 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35557.21 MB 2025-02-14 11:04:46,909 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:04:46,909 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30377.04 MB 2025-02-14 11:04:46,910 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 11:04:46,910 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 11:04:46,910 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.40 seconds 2025-02-14 11:04:46,910 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:04:46,910 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17501.49 MB 2025-02-14 11:04:46,910 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30343.85 MB 2025-02-14 11:04:46,910 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12842.35 MB 2025-02-14 11:04:46,910 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60301.51 MB 2025-02-14 11:04:46,910 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35557.21 MB 2025-02-14 11:04:46,910 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24744.30 MB 2025-02-14 11:04:46,910 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30377.04 MB 2025-02-14 11:04:47,178 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 11:04:47,178 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 11:04:47,178 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 11:04:47,178 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:04:47,178 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30343.85 MB 2025-02-14 11:04:47,178 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22489.50 MB 2025-02-14 11:04:47,178 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7854.34 MB 2025-02-14 11:04:47,178 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35557.21 MB 2025-02-14 11:04:47,178 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35557.21 MB 2025-02-14 11:04:47,178 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:04:47,178 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32842.30 MB 2025-02-14 11:04:47,195 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8119, cut from 8121 2025-02-14 11:04:47,196 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 11:04:47,201 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 11:04:47,201 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 11:04:47,201 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:04:47,201 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:04:47,201 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22489.50 MB 2025-02-14 11:04:47,201 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30884.72 MB 2025-02-14 11:04:47,201 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8395.21 MB 2025-02-14 11:04:47,202 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35557.21 MB 2025-02-14 11:04:47,202 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39730.54 MB 2025-02-14 11:04:47,202 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4173.33 MB 2025-02-14 11:04:47,202 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30884.72 MB 2025-02-14 11:04:47,363 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7911] 2025-02-14 11:04:47,364 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:04:47,364 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 11:04:47,365 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:04:47,365 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 11:04:47,370 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 11:04:47,371 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:04:47,371 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 11:04:47,371 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 11:05:41,575 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:05:41,576 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 11:05:41,581 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 11:05:41,584 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:05:41,584 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1312, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 11:05:41,585 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:05:41,585 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1312, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 11:06:01,765 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 11:06:01,765 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 11:06:01,765 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.17 seconds 2025-02-14 11:06:01,765 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:06:01,765 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22110.93 MB 2025-02-14 11:06:01,765 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26754.03 MB 2025-02-14 11:06:01,765 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4643.09 MB 2025-02-14 11:06:01,765 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48077.21 MB 2025-02-14 11:06:01,765 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38260.44 MB 2025-02-14 11:06:01,765 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9816.77 MB 2025-02-14 11:06:01,765 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35659.17 MB 2025-02-14 11:06:01,840 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 11:06:01,840 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 11:06:01,840 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 11:06:01,840 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:06:01,840 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26754.03 MB 2025-02-14 11:06:01,840 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22598.53 MB 2025-02-14 11:06:01,840 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4155.50 MB 2025-02-14 11:06:01,840 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38260.44 MB 2025-02-14 11:06:01,840 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47041.22 MB 2025-02-14 11:06:01,840 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8780.78 MB 2025-02-14 11:06:01,840 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40014.62 MB 2025-02-14 11:06:03,746 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 11:06:03,746 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 11:06:03,746 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.90 seconds 2025-02-14 11:06:03,746 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:06:03,746 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22598.53 MB 2025-02-14 11:06:03,746 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23129.37 MB 2025-02-14 11:06:03,746 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 11:06:03,746 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47041.22 MB 2025-02-14 11:06:03,746 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33617.35 MB 2025-02-14 11:06:03,746 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13423.87 MB 2025-02-14 11:06:03,746 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27108.70 MB 2025-02-14 11:06:03,759 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 11:06:03,759 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 11:06:03,759 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:06:03,759 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:06:03,759 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23129.37 MB 2025-02-14 11:06:03,760 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25018.90 MB 2025-02-14 11:06:03,760 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 11:06:03,760 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33617.35 MB 2025-02-14 11:06:03,760 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33617.35 MB 2025-02-14 11:06:03,760 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:06:03,760 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26436.33 MB 2025-02-14 11:06:03,971 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 11:06:03,971 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 11:06:03,971 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 11:06:03,971 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:06:03,971 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25018.90 MB 2025-02-14 11:06:03,971 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27260.76 MB 2025-02-14 11:06:03,971 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 11:06:03,971 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33617.35 MB 2025-02-14 11:06:03,971 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35976.64 MB 2025-02-14 11:06:03,971 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2359.30 MB 2025-02-14 11:06:03,971 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32805.04 MB 2025-02-14 11:06:03,972 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 11:06:03,972 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 11:06:03,972 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 11:06:03,972 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:06:03,972 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23129.37 MB 2025-02-14 11:06:03,972 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27260.76 MB 2025-02-14 11:06:03,972 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 11:06:03,972 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33617.35 MB 2025-02-14 11:06:03,972 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35976.64 MB 2025-02-14 11:06:03,972 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2359.30 MB 2025-02-14 11:06:03,972 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32805.04 MB 2025-02-14 11:06:04,141 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 11:06:04,141 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 11:06:04,141 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 11:06:04,141 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:06:04,141 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28794.30 MB 2025-02-14 11:06:04,141 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29561.30 MB 2025-02-14 11:06:04,141 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 11:06:04,141 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35976.64 MB 2025-02-14 11:06:04,141 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36391.88 MB 2025-02-14 11:06:04,141 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 11:06:04,141 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30269.09 MB 2025-02-14 11:06:04,160 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 11:06:04,160 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 11:06:04,160 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:06:04,160 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:06:04,160 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29974.19 MB 2025-02-14 11:06:04,160 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30202.32 MB 2025-02-14 11:06:04,160 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.13 MB 2025-02-14 11:06:04,160 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36391.88 MB 2025-02-14 11:06:04,160 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36391.88 MB 2025-02-14 11:06:04,160 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:06:04,160 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30415.63 MB 2025-02-14 11:06:04,161 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 11:06:04,161 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 11:06:04,161 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.57 seconds 2025-02-14 11:06:04,161 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:06:04,161 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17539.82 MB 2025-02-14 11:06:04,161 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30402.36 MB 2025-02-14 11:06:04,161 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12862.54 MB 2025-02-14 11:06:04,161 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48077.21 MB 2025-02-14 11:06:04,161 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36391.88 MB 2025-02-14 11:06:04,161 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11685.33 MB 2025-02-14 11:06:04,161 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30415.63 MB 2025-02-14 11:06:04,429 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 11:06:04,429 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 11:06:04,429 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 11:06:04,429 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:06:04,429 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30402.36 MB 2025-02-14 11:06:04,429 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22528.21 MB 2025-02-14 11:06:04,429 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7874.15 MB 2025-02-14 11:06:04,429 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36391.88 MB 2025-02-14 11:06:04,429 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36391.88 MB 2025-02-14 11:06:04,429 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:06:04,429 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32901.12 MB 2025-02-14 11:06:04,447 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8120, cut from 8122 2025-02-14 11:06:04,447 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 11:06:04,453 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 11:06:04,453 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 11:06:04,453 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:06:04,453 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:06:04,453 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22528.21 MB 2025-02-14 11:06:04,453 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30924.46 MB 2025-02-14 11:06:04,453 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8396.25 MB 2025-02-14 11:06:04,453 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36391.88 MB 2025-02-14 11:06:04,453 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40565.21 MB 2025-02-14 11:06:04,453 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4173.33 MB 2025-02-14 11:06:04,453 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30924.46 MB 2025-02-14 11:06:04,616 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7912] 2025-02-14 11:06:04,617 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:06:04,617 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 11:06:04,618 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:06:04,618 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 11:06:04,623 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 11:06:04,624 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:06:04,624 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 11:06:04,624 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 11:06:16,576 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:06:16,576 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 11:06:16,581 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 11:06:16,584 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:06:16,584 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1150, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 11:06:16,585 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:06:16,585 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1150, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 11:06:34,402 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 11:06:34,403 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 11:06:34,403 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.81 seconds 2025-02-14 11:06:34,403 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:06:34,403 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20982.09 MB 2025-02-14 11:06:34,403 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25052.66 MB 2025-02-14 11:06:34,403 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4070.57 MB 2025-02-14 11:06:34,403 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48911.88 MB 2025-02-14 11:06:34,403 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29324.48 MB 2025-02-14 11:06:34,403 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19587.40 MB 2025-02-14 11:06:34,403 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33851.65 MB 2025-02-14 11:06:34,479 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 11:06:34,479 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 11:06:34,479 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 11:06:34,479 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:06:34,479 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25052.66 MB 2025-02-14 11:06:34,479 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21756.34 MB 2025-02-14 11:06:34,479 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3296.32 MB 2025-02-14 11:06:34,479 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29324.48 MB 2025-02-14 11:06:34,479 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44293.95 MB 2025-02-14 11:06:34,479 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 14969.47 MB 2025-02-14 11:06:34,479 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37355.74 MB 2025-02-14 11:06:36,415 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 11:06:36,415 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 11:06:36,415 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 11:06:36,415 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:06:36,415 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21756.34 MB 2025-02-14 11:06:36,415 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22287.18 MB 2025-02-14 11:06:36,415 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 11:06:36,415 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44293.95 MB 2025-02-14 11:06:36,415 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26669.48 MB 2025-02-14 11:06:36,415 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17624.47 MB 2025-02-14 11:06:36,415 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26266.52 MB 2025-02-14 11:06:36,429 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 11:06:36,429 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 11:06:36,429 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:06:36,429 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:06:36,429 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22287.18 MB 2025-02-14 11:06:36,429 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24176.72 MB 2025-02-14 11:06:36,429 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 11:06:36,429 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26669.48 MB 2025-02-14 11:06:36,429 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28556.92 MB 2025-02-14 11:06:36,429 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 11:06:36,429 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25594.15 MB 2025-02-14 11:06:36,639 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 11:06:36,639 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 11:06:36,639 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 11:06:36,639 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:06:36,639 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24176.72 MB 2025-02-14 11:06:36,639 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26418.57 MB 2025-02-14 11:06:36,639 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 11:06:36,639 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28556.92 MB 2025-02-14 11:06:36,639 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34219.23 MB 2025-02-14 11:06:36,639 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 11:06:36,639 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31962.86 MB 2025-02-14 11:06:36,640 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 11:06:36,640 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 11:06:36,640 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 11:06:36,640 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:06:36,640 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22287.18 MB 2025-02-14 11:06:36,640 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26418.57 MB 2025-02-14 11:06:36,640 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 11:06:36,640 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26669.48 MB 2025-02-14 11:06:36,640 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34219.23 MB 2025-02-14 11:06:36,640 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-14 11:06:36,640 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31962.86 MB 2025-02-14 11:06:36,805 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 11:06:36,806 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 11:06:36,806 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 11:06:36,806 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:06:36,806 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27952.12 MB 2025-02-14 11:06:36,806 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28719.12 MB 2025-02-14 11:06:36,806 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 11:06:36,806 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34219.23 MB 2025-02-14 11:06:36,806 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34636.56 MB 2025-02-14 11:06:36,806 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 11:06:36,806 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29426.91 MB 2025-02-14 11:06:36,824 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 11:06:36,824 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 11:06:36,824 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:06:36,824 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:06:36,824 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29132.01 MB 2025-02-14 11:06:36,824 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29360.57 MB 2025-02-14 11:06:36,824 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.57 MB 2025-02-14 11:06:36,824 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34636.56 MB 2025-02-14 11:06:36,824 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34636.56 MB 2025-02-14 11:06:36,824 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:06:36,824 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29594.37 MB 2025-02-14 11:06:36,825 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 11:06:36,825 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 11:06:36,825 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.24 seconds 2025-02-14 11:06:36,825 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:06:36,825 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16975.40 MB 2025-02-14 11:06:36,825 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29561.06 MB 2025-02-14 11:06:36,825 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12585.66 MB 2025-02-14 11:06:36,825 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48911.88 MB 2025-02-14 11:06:36,825 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34636.56 MB 2025-02-14 11:06:36,825 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14275.31 MB 2025-02-14 11:06:36,825 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29594.37 MB 2025-02-14 11:06:37,095 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 11:06:37,095 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 11:06:37,095 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 11:06:37,095 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:06:37,095 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29561.06 MB 2025-02-14 11:06:37,095 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21970.65 MB 2025-02-14 11:06:37,095 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7590.41 MB 2025-02-14 11:06:37,095 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34636.56 MB 2025-02-14 11:06:37,095 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34636.56 MB 2025-02-14 11:06:37,095 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:06:37,095 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32065.35 MB 2025-02-14 11:06:37,113 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8138, cut from 8140 2025-02-14 11:06:37,113 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 11:06:37,119 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 11:06:37,119 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 11:06:37,119 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:06:37,119 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:06:37,119 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21970.65 MB 2025-02-14 11:06:37,119 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30384.63 MB 2025-02-14 11:06:37,119 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8413.98 MB 2025-02-14 11:06:37,119 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34636.56 MB 2025-02-14 11:06:37,119 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43002.10 MB 2025-02-14 11:06:37,119 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8365.54 MB 2025-02-14 11:06:37,119 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30384.63 MB 2025-02-14 11:06:37,282 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7930] 2025-02-14 11:06:37,283 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:06:37,283 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 11:06:37,284 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:06:37,284 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 11:06:37,289 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 11:06:37,290 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:06:37,290 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 11:06:37,290 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 11:07:37,972 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:07:37,972 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 11:07:37,977 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 11:07:37,981 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:07:37,981 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 212, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 11:07:37,982 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:07:37,982 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 212, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 11:07:41,273 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 11:07:41,273 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 11:07:41,273 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.29 seconds 2025-02-14 11:07:41,273 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:07:41,273 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14445.96 MB 2025-02-14 11:07:41,273 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15196.21 MB 2025-02-14 11:07:41,273 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 750.26 MB 2025-02-14 11:07:41,273 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55549.36 MB 2025-02-14 11:07:41,273 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17853.05 MB 2025-02-14 11:07:41,273 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -37696.31 MB 2025-02-14 11:07:41,273 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24144.63 MB 2025-02-14 11:07:41,287 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 11:07:41,287 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 11:07:41,287 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:07:41,287 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:07:41,287 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15196.21 MB 2025-02-14 11:07:41,287 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15433.29 MB 2025-02-14 11:07:41,287 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 237.08 MB 2025-02-14 11:07:41,287 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17853.05 MB 2025-02-14 11:07:41,287 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19228.79 MB 2025-02-14 11:07:41,287 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1375.73 MB 2025-02-14 11:07:41,287 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17936.02 MB 2025-02-14 11:07:42,224 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 11:07:42,224 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 11:07:42,224 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.94 seconds 2025-02-14 11:07:42,224 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:07:42,224 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15433.29 MB 2025-02-14 11:07:42,224 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15690.75 MB 2025-02-14 11:07:42,224 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 257.46 MB 2025-02-14 11:07:42,224 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19228.79 MB 2025-02-14 11:07:42,224 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18943.57 MB 2025-02-14 11:07:42,224 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -285.21 MB 2025-02-14 11:07:42,225 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19690.57 MB 2025-02-14 11:07:42,233 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 11:07:42,233 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 11:07:42,233 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:07:42,233 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:07:42,233 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15690.69 MB 2025-02-14 11:07:42,233 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16606.89 MB 2025-02-14 11:07:42,233 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 916.20 MB 2025-02-14 11:07:42,233 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18943.57 MB 2025-02-14 11:07:42,233 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19402.85 MB 2025-02-14 11:07:42,233 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 459.28 MB 2025-02-14 11:07:42,233 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17294.35 MB 2025-02-14 11:07:42,338 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 11:07:42,338 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 11:07:42,338 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 11:07:42,338 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:07:42,338 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16606.89 MB 2025-02-14 11:07:42,338 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17694.42 MB 2025-02-14 11:07:42,338 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1087.53 MB 2025-02-14 11:07:42,338 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19402.85 MB 2025-02-14 11:07:42,338 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22158.51 MB 2025-02-14 11:07:42,338 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2755.66 MB 2025-02-14 11:07:42,338 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20383.36 MB 2025-02-14 11:07:42,339 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 11:07:42,339 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 11:07:42,339 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 11:07:42,339 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:07:42,339 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15690.69 MB 2025-02-14 11:07:42,339 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17694.42 MB 2025-02-14 11:07:42,339 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2003.73 MB 2025-02-14 11:07:42,339 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18943.57 MB 2025-02-14 11:07:42,339 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22158.51 MB 2025-02-14 11:07:42,339 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3214.93 MB 2025-02-14 11:07:42,339 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20383.36 MB 2025-02-14 11:07:42,424 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 11:07:42,424 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 11:07:42,424 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 11:07:42,424 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:07:42,424 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18438.19 MB 2025-02-14 11:07:42,424 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18810.18 MB 2025-02-14 11:07:42,424 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 372.00 MB 2025-02-14 11:07:42,424 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22158.51 MB 2025-02-14 11:07:42,424 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22357.74 MB 2025-02-14 11:07:42,424 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 199.23 MB 2025-02-14 11:07:42,424 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19157.55 MB 2025-02-14 11:07:42,436 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 11:07:42,436 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 11:07:42,436 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:07:42,436 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:07:42,436 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19010.44 MB 2025-02-14 11:07:42,436 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19215.09 MB 2025-02-14 11:07:42,436 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 204.64 MB 2025-02-14 11:07:42,436 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22357.74 MB 2025-02-14 11:07:42,436 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22361.93 MB 2025-02-14 11:07:42,436 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-14 11:07:42,436 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19266.72 MB 2025-02-14 11:07:42,437 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 11:07:42,437 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 11:07:42,437 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.45 seconds 2025-02-14 11:07:42,437 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:07:42,437 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13707.33 MB 2025-02-14 11:07:42,437 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19416.16 MB 2025-02-14 11:07:42,437 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5708.83 MB 2025-02-14 11:07:42,437 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55549.36 MB 2025-02-14 11:07:42,437 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22361.93 MB 2025-02-14 11:07:42,437 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33187.43 MB 2025-02-14 11:07:42,437 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19416.16 MB 2025-02-14 11:07:42,703 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 11:07:42,703 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 11:07:42,703 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 11:07:42,703 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:07:42,703 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14724.99 MB 2025-02-14 11:07:42,703 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17739.02 MB 2025-02-14 11:07:42,703 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-14 11:07:42,703 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22361.93 MB 2025-02-14 11:07:42,703 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22361.93 MB 2025-02-14 11:07:42,703 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:07:42,703 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18040.39 MB 2025-02-14 11:07:42,721 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 11:07:42,722 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 11:07:42,728 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 11:07:42,728 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 11:07:42,728 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:07:42,728 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:07:42,728 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17739.02 MB 2025-02-14 11:07:42,728 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26178.05 MB 2025-02-14 11:07:42,728 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 11:07:42,728 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22361.93 MB 2025-02-14 11:07:42,728 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32851.89 MB 2025-02-14 11:07:42,728 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 11:07:42,728 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26178.05 MB 2025-02-14 11:07:42,892 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 11:07:42,893 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:07:42,893 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 11:07:42,894 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:07:42,894 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 11:07:42,899 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 11:07:42,900 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:07:42,900 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 11:07:42,900 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 11:08:31,922 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:08:31,923 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 11:08:31,928 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 11:08:31,931 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:08:31,931 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1434, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 11:08:31,932 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:08:31,932 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1434, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 11:08:53,838 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 11:08:53,838 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 11:08:53,838 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.90 seconds 2025-02-14 11:08:53,838 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:08:53,838 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22961.05 MB 2025-02-14 11:08:53,838 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28036.16 MB 2025-02-14 11:08:53,838 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5075.11 MB 2025-02-14 11:08:53,838 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45436.90 MB 2025-02-14 11:08:53,838 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38761.66 MB 2025-02-14 11:08:53,838 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6675.23 MB 2025-02-14 11:08:53,838 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36962.27 MB 2025-02-14 11:08:53,922 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 11:08:53,922 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 11:08:53,922 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 11:08:53,922 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:08:53,922 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28036.16 MB 2025-02-14 11:08:53,922 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23232.77 MB 2025-02-14 11:08:53,922 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4803.39 MB 2025-02-14 11:08:53,922 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38761.66 MB 2025-02-14 11:08:53,922 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48639.25 MB 2025-02-14 11:08:53,922 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9877.59 MB 2025-02-14 11:08:53,922 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42924.15 MB 2025-02-14 11:08:55,835 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 11:08:55,835 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 11:08:55,835 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 11:08:55,835 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:08:55,835 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23232.77 MB 2025-02-14 11:08:55,835 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23763.61 MB 2025-02-14 11:08:55,835 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 11:08:55,835 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48639.25 MB 2025-02-14 11:08:55,835 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33686.55 MB 2025-02-14 11:08:55,835 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14952.69 MB 2025-02-14 11:08:55,835 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27742.94 MB 2025-02-14 11:08:55,848 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 11:08:55,848 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 11:08:55,848 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:08:55,848 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:08:55,848 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23763.61 MB 2025-02-14 11:08:55,849 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25653.14 MB 2025-02-14 11:08:55,849 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 11:08:55,849 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33686.55 MB 2025-02-14 11:08:55,849 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33686.55 MB 2025-02-14 11:08:55,849 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:08:55,849 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27070.57 MB 2025-02-14 11:08:56,062 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 11:08:56,062 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 11:08:56,062 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 11:08:56,062 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:08:56,062 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25653.14 MB 2025-02-14 11:08:56,062 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27895.00 MB 2025-02-14 11:08:56,062 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 11:08:56,062 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33686.55 MB 2025-02-14 11:08:56,062 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37461.43 MB 2025-02-14 11:08:56,062 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 11:08:56,062 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33439.28 MB 2025-02-14 11:08:56,063 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 11:08:56,063 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 11:08:56,063 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 11:08:56,063 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:08:56,063 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23763.61 MB 2025-02-14 11:08:56,063 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27895.00 MB 2025-02-14 11:08:56,063 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 11:08:56,063 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33686.55 MB 2025-02-14 11:08:56,063 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37461.43 MB 2025-02-14 11:08:56,063 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 11:08:56,063 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33439.28 MB 2025-02-14 11:08:56,277 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 11:08:56,277 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 11:08:56,277 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 11:08:56,277 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:08:56,277 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29428.54 MB 2025-02-14 11:08:56,277 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30195.54 MB 2025-02-14 11:08:56,277 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 11:08:56,277 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37461.43 MB 2025-02-14 11:08:56,277 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37878.76 MB 2025-02-14 11:08:56,277 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 11:08:56,277 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30903.33 MB 2025-02-14 11:08:56,296 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 11:08:56,296 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 11:08:56,296 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:08:56,296 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:08:56,296 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30608.43 MB 2025-02-14 11:08:56,296 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30836.76 MB 2025-02-14 11:08:56,296 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.32 MB 2025-02-14 11:08:56,296 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37878.76 MB 2025-02-14 11:08:56,296 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37878.76 MB 2025-02-14 11:08:56,296 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:08:56,296 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31072.77 MB 2025-02-14 11:08:56,297 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 11:08:56,297 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 11:08:56,297 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.36 seconds 2025-02-14 11:08:56,297 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:08:56,297 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17964.88 MB 2025-02-14 11:08:56,297 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31036.99 MB 2025-02-14 11:08:56,297 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13072.12 MB 2025-02-14 11:08:56,297 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45436.90 MB 2025-02-14 11:08:56,297 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37878.76 MB 2025-02-14 11:08:56,297 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7558.14 MB 2025-02-14 11:08:56,298 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31072.77 MB 2025-02-14 11:08:56,565 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 11:08:56,565 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 11:08:56,565 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 11:08:56,565 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:08:56,565 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31036.99 MB 2025-02-14 11:08:56,565 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22956.31 MB 2025-02-14 11:08:56,565 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8080.68 MB 2025-02-14 11:08:56,565 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37878.76 MB 2025-02-14 11:08:56,565 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37878.76 MB 2025-02-14 11:08:56,565 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:08:56,565 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33538.21 MB 2025-02-14 11:08:56,583 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8128, cut from 8130 2025-02-14 11:08:56,583 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 11:08:56,590 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 11:08:56,590 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 11:08:56,590 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:08:56,590 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:08:56,590 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22956.31 MB 2025-02-14 11:08:56,590 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31360.91 MB 2025-02-14 11:08:56,590 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8404.59 MB 2025-02-14 11:08:56,590 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37878.76 MB 2025-02-14 11:08:56,590 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42056.29 MB 2025-02-14 11:08:56,590 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4177.53 MB 2025-02-14 11:08:56,590 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31360.91 MB 2025-02-14 11:08:56,754 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7920] 2025-02-14 11:08:56,755 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:08:56,755 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 11:08:56,756 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:08:56,756 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 11:08:56,761 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 11:08:56,762 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:08:56,762 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 11:08:56,762 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 11:09:20,126 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:09:20,126 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 11:09:20,133 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 11:09:20,138 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:09:20,138 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1183, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 11:09:20,140 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:09:20,140 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1183, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 11:09:38,411 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 11:09:38,412 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 11:09:38,412 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.26 seconds 2025-02-14 11:09:38,412 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:09:38,412 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21212.04 MB 2025-02-14 11:09:38,412 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25398.61 MB 2025-02-14 11:09:38,412 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4186.57 MB 2025-02-14 11:09:38,412 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50411.34 MB 2025-02-14 11:09:38,412 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29471.28 MB 2025-02-14 11:09:38,412 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20940.06 MB 2025-02-14 11:09:38,412 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34308.10 MB 2025-02-14 11:09:38,491 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 11:09:38,491 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 11:09:38,491 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 11:09:38,491 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:09:38,491 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25398.61 MB 2025-02-14 11:09:38,491 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21927.90 MB 2025-02-14 11:09:38,491 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3470.71 MB 2025-02-14 11:09:38,491 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29471.28 MB 2025-02-14 11:09:38,491 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45067.80 MB 2025-02-14 11:09:38,491 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 15596.52 MB 2025-02-14 11:09:38,491 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37887.55 MB 2025-02-14 11:09:40,412 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 11:09:40,412 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 11:09:40,413 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 11:09:40,413 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:09:40,413 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21927.90 MB 2025-02-14 11:09:40,413 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22458.74 MB 2025-02-14 11:09:40,413 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 11:09:40,413 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45067.80 MB 2025-02-14 11:09:40,413 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26698.84 MB 2025-02-14 11:09:40,413 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18368.95 MB 2025-02-14 11:09:40,413 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26438.07 MB 2025-02-14 11:09:40,426 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 11:09:40,426 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 11:09:40,426 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:09:40,426 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:09:40,426 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22458.74 MB 2025-02-14 11:09:40,426 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24348.27 MB 2025-02-14 11:09:40,426 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 11:09:40,426 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26698.84 MB 2025-02-14 11:09:40,426 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28586.28 MB 2025-02-14 11:09:40,426 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 11:09:40,427 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25765.70 MB 2025-02-14 11:09:40,636 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 11:09:40,636 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 11:09:40,636 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 11:09:40,636 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:09:40,636 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24348.27 MB 2025-02-14 11:09:40,636 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26590.13 MB 2025-02-14 11:09:40,636 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 11:09:40,636 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28586.28 MB 2025-02-14 11:09:40,636 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34248.59 MB 2025-02-14 11:09:40,636 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 11:09:40,636 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32134.41 MB 2025-02-14 11:09:40,637 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 11:09:40,637 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 11:09:40,637 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 11:09:40,637 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:09:40,637 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22458.74 MB 2025-02-14 11:09:40,637 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26590.13 MB 2025-02-14 11:09:40,637 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 11:09:40,637 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26698.84 MB 2025-02-14 11:09:40,637 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34248.59 MB 2025-02-14 11:09:40,637 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-14 11:09:40,637 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32134.41 MB 2025-02-14 11:09:40,810 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 11:09:40,810 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 11:09:40,810 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 11:09:40,810 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:09:40,810 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28123.67 MB 2025-02-14 11:09:40,810 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28890.67 MB 2025-02-14 11:09:40,810 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 11:09:40,810 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34248.59 MB 2025-02-14 11:09:40,810 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34663.83 MB 2025-02-14 11:09:40,810 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 11:09:40,810 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29598.46 MB 2025-02-14 11:09:40,829 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 11:09:40,829 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 11:09:40,829 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:09:40,829 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:09:40,829 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29303.56 MB 2025-02-14 11:09:40,829 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29532.72 MB 2025-02-14 11:09:40,829 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.16 MB 2025-02-14 11:09:40,829 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34663.83 MB 2025-02-14 11:09:40,829 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34663.83 MB 2025-02-14 11:09:40,829 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:09:40,829 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29767.73 MB 2025-02-14 11:09:40,830 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 11:09:40,830 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 11:09:40,830 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.69 seconds 2025-02-14 11:09:40,830 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:09:40,830 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17090.37 MB 2025-02-14 11:09:40,830 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29733.79 MB 2025-02-14 11:09:40,830 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12643.42 MB 2025-02-14 11:09:40,830 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50411.34 MB 2025-02-14 11:09:40,830 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34663.83 MB 2025-02-14 11:09:40,830 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15747.51 MB 2025-02-14 11:09:40,830 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29767.73 MB 2025-02-14 11:09:41,099 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 11:09:41,099 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 11:09:41,099 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 11:09:41,100 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:09:41,100 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29733.79 MB 2025-02-14 11:09:41,100 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22094.76 MB 2025-02-14 11:09:41,100 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7639.03 MB 2025-02-14 11:09:41,100 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34663.83 MB 2025-02-14 11:09:41,100 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34663.83 MB 2025-02-14 11:09:41,100 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:09:41,100 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32245.46 MB 2025-02-14 11:09:41,117 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 11:09:41,118 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 11:09:41,124 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 11:09:41,124 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 11:09:41,124 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:09:41,124 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:09:41,124 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22094.76 MB 2025-02-14 11:09:41,124 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30533.79 MB 2025-02-14 11:09:41,124 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 11:09:41,124 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34663.83 MB 2025-02-14 11:09:41,124 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43054.53 MB 2025-02-14 11:09:41,124 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 11:09:41,124 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30533.79 MB 2025-02-14 11:09:41,289 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 11:09:41,291 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:09:41,291 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 11:09:41,292 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:09:41,292 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 11:09:41,296 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 11:09:41,297 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:09:41,297 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 11:09:41,298 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 11:11:03,826 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:11:03,826 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 11:11:03,835 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 11:11:03,842 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:11:03,842 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 451, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 11:11:03,844 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:11:03,844 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 451, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 11:11:10,913 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 11:11:10,913 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 11:11:10,913 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.06 seconds 2025-02-14 11:11:10,913 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:11:10,913 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16111.35 MB 2025-02-14 11:11:10,913 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17707.41 MB 2025-02-14 11:11:10,913 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1596.06 MB 2025-02-14 11:11:10,913 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55639.54 MB 2025-02-14 11:11:10,913 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21338.52 MB 2025-02-14 11:11:10,913 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34301.02 MB 2025-02-14 11:11:10,913 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26715.99 MB 2025-02-14 11:11:10,946 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 11:11:10,946 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 11:11:10,946 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 11:11:10,946 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:11:10,946 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17707.41 MB 2025-02-14 11:11:10,946 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18123.51 MB 2025-02-14 11:11:10,946 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 416.10 MB 2025-02-14 11:11:10,946 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21338.52 MB 2025-02-14 11:11:10,946 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28034.73 MB 2025-02-14 11:11:10,946 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6696.21 MB 2025-02-14 11:11:10,946 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25098.84 MB 2025-02-14 11:11:12,849 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 11:11:12,849 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 11:11:12,849 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.90 seconds 2025-02-14 11:11:12,849 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:11:12,849 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18123.51 MB 2025-02-14 11:11:12,849 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18654.35 MB 2025-02-14 11:11:12,849 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 11:11:12,849 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28034.73 MB 2025-02-14 11:11:12,849 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20921.19 MB 2025-02-14 11:11:12,849 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7113.54 MB 2025-02-14 11:11:12,849 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22634.72 MB 2025-02-14 11:11:12,863 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 11:11:12,863 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 11:11:12,863 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:11:12,863 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:11:12,863 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18654.35 MB 2025-02-14 11:11:12,863 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20543.88 MB 2025-02-14 11:11:12,863 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 11:11:12,863 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20921.19 MB 2025-02-14 11:11:12,863 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24224.20 MB 2025-02-14 11:11:12,863 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 11:11:12,863 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21961.31 MB 2025-02-14 11:11:13,076 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 11:11:13,076 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 11:11:13,076 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 11:11:13,076 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:11:13,076 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20543.88 MB 2025-02-14 11:11:13,076 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22785.74 MB 2025-02-14 11:11:13,076 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 11:11:13,076 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24224.20 MB 2025-02-14 11:11:13,076 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30830.23 MB 2025-02-14 11:11:13,076 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 11:11:13,076 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28330.02 MB 2025-02-14 11:11:13,076 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 11:11:13,077 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 11:11:13,077 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 11:11:13,077 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:11:13,077 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18654.35 MB 2025-02-14 11:11:13,077 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22785.74 MB 2025-02-14 11:11:13,077 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 11:11:13,077 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20921.19 MB 2025-02-14 11:11:13,077 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30830.23 MB 2025-02-14 11:11:13,077 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9909.04 MB 2025-02-14 11:11:13,077 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28330.02 MB 2025-02-14 11:11:13,247 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 11:11:13,247 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 11:11:13,247 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 11:11:13,247 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:11:13,247 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24319.28 MB 2025-02-14 11:11:13,247 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25086.28 MB 2025-02-14 11:11:13,247 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 11:11:13,247 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30830.23 MB 2025-02-14 11:11:13,247 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31245.47 MB 2025-02-14 11:11:13,247 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 11:11:13,247 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25794.07 MB 2025-02-14 11:11:13,272 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 11:11:13,272 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 11:11:13,272 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:11:13,273 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:11:13,273 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25499.17 MB 2025-02-14 11:11:13,273 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25727.80 MB 2025-02-14 11:11:13,273 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.63 MB 2025-02-14 11:11:13,273 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31245.47 MB 2025-02-14 11:11:13,273 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31245.47 MB 2025-02-14 11:11:13,273 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:11:13,273 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25938.16 MB 2025-02-14 11:11:13,277 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 11:11:13,277 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 11:11:13,277 - resource_logging.py:150 - __exit__ - DEBUG - Time: 9.43 seconds 2025-02-14 11:11:13,277 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:11:13,277 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14540.03 MB 2025-02-14 11:11:13,277 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25928.87 MB 2025-02-14 11:11:13,277 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11388.85 MB 2025-02-14 11:11:13,277 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55639.54 MB 2025-02-14 11:11:13,277 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31245.47 MB 2025-02-14 11:11:13,277 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24394.07 MB 2025-02-14 11:11:13,277 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25938.16 MB 2025-02-14 11:11:13,561 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 11:11:13,561 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 11:11:13,561 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.28 seconds 2025-02-14 11:11:13,561 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:11:13,561 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25928.87 MB 2025-02-14 11:11:13,561 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19544.42 MB 2025-02-14 11:11:13,561 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6384.46 MB 2025-02-14 11:11:13,561 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31245.47 MB 2025-02-14 11:11:13,561 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31245.47 MB 2025-02-14 11:11:13,561 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:11:13,561 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28440.54 MB 2025-02-14 11:11:13,580 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 11:11:13,581 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 11:11:13,588 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 11:11:13,588 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 11:11:13,588 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:11:13,588 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:11:13,588 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19544.42 MB 2025-02-14 11:11:13,588 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27983.44 MB 2025-02-14 11:11:13,588 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 11:11:13,588 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31245.47 MB 2025-02-14 11:11:13,588 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41735.42 MB 2025-02-14 11:11:13,588 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 11:11:13,588 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27983.44 MB 2025-02-14 11:11:13,826 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 11:11:13,829 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:11:13,829 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 11:11:13,831 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:11:13,831 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 11:11:13,838 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 11:11:13,840 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:11:13,840 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 11:11:13,840 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 11:12:42,896 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:12:42,896 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 11:12:42,901 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 11:12:42,905 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:12:42,905 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1737, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 11:12:42,906 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:12:42,906 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1737, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 11:13:09,532 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 11:13:09,532 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 11:13:09,532 - resource_logging.py:150 - __exit__ - DEBUG - Time: 26.62 seconds 2025-02-14 11:13:09,532 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:13:09,532 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25072.40 MB 2025-02-14 11:13:09,532 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31219.55 MB 2025-02-14 11:13:09,532 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6147.15 MB 2025-02-14 11:13:09,532 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54320.43 MB 2025-02-14 11:13:09,532 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39835.40 MB 2025-02-14 11:13:09,532 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14485.03 MB 2025-02-14 11:13:09,532 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40206.89 MB 2025-02-14 11:13:09,635 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 11:13:09,635 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 11:13:09,635 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 11:13:09,635 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:13:09,635 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31219.55 MB 2025-02-14 11:13:09,635 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24807.97 MB 2025-02-14 11:13:09,635 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6411.57 MB 2025-02-14 11:13:09,635 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39835.40 MB 2025-02-14 11:13:09,635 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56904.12 MB 2025-02-14 11:13:09,635 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 17068.72 MB 2025-02-14 11:13:09,635 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48228.35 MB 2025-02-14 11:13:11,545 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 11:13:11,545 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 11:13:11,545 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 11:13:11,546 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:13:11,546 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24807.97 MB 2025-02-14 11:13:11,546 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25338.81 MB 2025-02-14 11:13:11,546 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 11:13:11,546 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56904.12 MB 2025-02-14 11:13:11,546 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35102.13 MB 2025-02-14 11:13:11,546 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21801.99 MB 2025-02-14 11:13:11,546 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29318.15 MB 2025-02-14 11:13:11,559 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 11:13:11,559 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 11:13:11,559 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:13:11,559 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:13:11,559 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25338.81 MB 2025-02-14 11:13:11,559 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27228.35 MB 2025-02-14 11:13:11,559 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 11:13:11,559 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35102.13 MB 2025-02-14 11:13:11,559 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35102.13 MB 2025-02-14 11:13:11,559 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:13:11,559 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28645.78 MB 2025-02-14 11:13:11,772 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 11:13:11,772 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 11:13:11,772 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 11:13:11,772 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:13:11,772 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27228.35 MB 2025-02-14 11:13:11,772 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29470.20 MB 2025-02-14 11:13:11,772 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 11:13:11,772 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35102.13 MB 2025-02-14 11:13:11,772 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38405.14 MB 2025-02-14 11:13:11,772 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 11:13:11,772 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35014.48 MB 2025-02-14 11:13:11,773 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 11:13:11,773 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 11:13:11,773 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 11:13:11,773 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:13:11,773 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25338.81 MB 2025-02-14 11:13:11,773 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29470.20 MB 2025-02-14 11:13:11,773 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 11:13:11,773 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35102.13 MB 2025-02-14 11:13:11,773 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38405.14 MB 2025-02-14 11:13:11,773 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 11:13:11,773 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35014.48 MB 2025-02-14 11:13:11,996 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 11:13:11,996 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 11:13:11,996 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 11:13:11,996 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:13:11,996 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31003.75 MB 2025-02-14 11:13:11,996 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31770.75 MB 2025-02-14 11:13:11,996 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 11:13:11,996 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38405.14 MB 2025-02-14 11:13:11,996 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38822.48 MB 2025-02-14 11:13:11,996 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 11:13:11,996 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32478.54 MB 2025-02-14 11:13:12,015 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 11:13:12,015 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 11:13:12,015 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:13:12,015 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:13:12,015 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32183.70 MB 2025-02-14 11:13:12,015 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32412.27 MB 2025-02-14 11:13:12,015 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.56 MB 2025-02-14 11:13:12,015 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38822.48 MB 2025-02-14 11:13:12,015 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38822.48 MB 2025-02-14 11:13:12,015 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:13:12,015 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32628.30 MB 2025-02-14 11:13:12,016 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 11:13:12,016 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 11:13:12,016 - resource_logging.py:150 - __exit__ - DEBUG - Time: 29.11 seconds 2025-02-14 11:13:12,016 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:13:12,016 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19020.55 MB 2025-02-14 11:13:12,016 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32612.60 MB 2025-02-14 11:13:12,016 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13592.05 MB 2025-02-14 11:13:12,016 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54320.43 MB 2025-02-14 11:13:12,016 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38822.48 MB 2025-02-14 11:13:12,016 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15497.95 MB 2025-02-14 11:13:12,016 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32628.30 MB 2025-02-14 11:13:12,286 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 11:13:12,286 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 11:13:12,286 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 11:13:12,286 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:13:12,286 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32612.60 MB 2025-02-14 11:13:12,286 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24013.51 MB 2025-02-14 11:13:12,286 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8599.09 MB 2025-02-14 11:13:12,286 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38822.48 MB 2025-02-14 11:13:12,286 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38822.48 MB 2025-02-14 11:13:12,286 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:13:12,286 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35115.05 MB 2025-02-14 11:13:12,304 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8132, cut from 8134 2025-02-14 11:13:12,304 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 11:13:12,310 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 11:13:12,310 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 11:13:12,310 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:13:12,310 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:13:12,310 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24013.51 MB 2025-02-14 11:13:12,310 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32421.24 MB 2025-02-14 11:13:12,310 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8407.72 MB 2025-02-14 11:13:12,310 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38822.48 MB 2025-02-14 11:13:12,310 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38822.48 MB 2025-02-14 11:13:12,310 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:13:12,310 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32421.24 MB 2025-02-14 11:13:12,472 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7924] 2025-02-14 11:13:12,474 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:13:12,474 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 11:13:12,475 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:13:12,475 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 11:13:12,479 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 11:13:12,480 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:13:12,480 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 11:13:12,481 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 11:13:20,910 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:13:20,910 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 11:13:20,915 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 11:13:20,918 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:13:20,918 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2004, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 11:13:20,919 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:13:20,919 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2004, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 11:13:52,035 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 11:13:52,035 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 11:13:52,035 - resource_logging.py:150 - __exit__ - DEBUG - Time: 31.11 seconds 2025-02-14 11:13:52,035 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:13:52,035 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26932.90 MB 2025-02-14 11:13:52,035 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34025.47 MB 2025-02-14 11:13:52,035 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7092.57 MB 2025-02-14 11:13:52,035 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47181.73 MB 2025-02-14 11:13:52,035 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40749.76 MB 2025-02-14 11:13:52,035 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6431.97 MB 2025-02-14 11:13:52,035 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42973.36 MB 2025-02-14 11:13:52,158 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 11:13:52,158 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 11:13:52,158 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 11:13:52,158 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:13:52,158 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34025.47 MB 2025-02-14 11:13:52,158 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26196.02 MB 2025-02-14 11:13:52,158 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7829.45 MB 2025-02-14 11:13:52,158 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40749.76 MB 2025-02-14 11:13:52,158 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 64495.81 MB 2025-02-14 11:13:52,158 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 23746.05 MB 2025-02-14 11:13:52,158 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54562.87 MB 2025-02-14 11:13:54,110 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 11:13:54,110 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 11:13:54,110 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.95 seconds 2025-02-14 11:13:54,110 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:13:54,110 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26196.02 MB 2025-02-14 11:13:54,110 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26726.86 MB 2025-02-14 11:13:54,110 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 11:13:54,110 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64495.81 MB 2025-02-14 11:13:54,110 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30878.47 MB 2025-02-14 11:13:54,110 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33617.35 MB 2025-02-14 11:13:54,110 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30707.23 MB 2025-02-14 11:13:54,123 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 11:13:54,123 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 11:13:54,123 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:13:54,123 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:13:54,124 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26726.86 MB 2025-02-14 11:13:54,124 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28616.40 MB 2025-02-14 11:13:54,124 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 11:13:54,124 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30878.47 MB 2025-02-14 11:13:54,124 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32765.90 MB 2025-02-14 11:13:54,124 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 11:13:54,124 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30033.83 MB 2025-02-14 11:13:54,334 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 11:13:54,334 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 11:13:54,334 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 11:13:54,334 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:13:54,334 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28616.40 MB 2025-02-14 11:13:54,334 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30858.25 MB 2025-02-14 11:13:54,334 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 11:13:54,334 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32765.90 MB 2025-02-14 11:13:54,335 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38900.07 MB 2025-02-14 11:13:54,335 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 11:13:54,335 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36402.53 MB 2025-02-14 11:13:54,335 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 11:13:54,335 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 11:13:54,335 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 11:13:54,335 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:13:54,335 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26726.86 MB 2025-02-14 11:13:54,335 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30858.25 MB 2025-02-14 11:13:54,335 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 11:13:54,335 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30878.47 MB 2025-02-14 11:13:54,335 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38900.07 MB 2025-02-14 11:13:54,335 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-14 11:13:54,335 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36402.53 MB 2025-02-14 11:13:54,502 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 11:13:54,502 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 11:13:54,502 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 11:13:54,502 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:13:54,502 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32391.80 MB 2025-02-14 11:13:54,502 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33158.80 MB 2025-02-14 11:13:54,502 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 11:13:54,502 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38900.07 MB 2025-02-14 11:13:54,502 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39315.31 MB 2025-02-14 11:13:54,502 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 11:13:54,502 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33866.59 MB 2025-02-14 11:13:54,521 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 11:13:54,521 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 11:13:54,521 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:13:54,521 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:13:54,521 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33571.69 MB 2025-02-14 11:13:54,521 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33800.71 MB 2025-02-14 11:13:54,521 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.02 MB 2025-02-14 11:13:54,521 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39315.31 MB 2025-02-14 11:13:54,521 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39315.31 MB 2025-02-14 11:13:54,521 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:13:54,521 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34009.17 MB 2025-02-14 11:13:54,522 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 11:13:54,522 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 11:13:54,522 - resource_logging.py:150 - __exit__ - DEBUG - Time: 33.60 seconds 2025-02-14 11:13:54,522 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:13:54,522 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19950.80 MB 2025-02-14 11:13:54,522 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34000.77 MB 2025-02-14 11:13:54,522 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14049.97 MB 2025-02-14 11:13:54,522 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47181.73 MB 2025-02-14 11:13:54,522 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39315.31 MB 2025-02-14 11:13:54,522 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7866.42 MB 2025-02-14 11:13:54,522 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34009.17 MB 2025-02-14 11:13:54,792 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 11:13:54,792 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 11:13:54,792 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 11:13:54,792 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:13:54,792 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34000.77 MB 2025-02-14 11:13:54,792 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24939.57 MB 2025-02-14 11:13:54,792 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9061.20 MB 2025-02-14 11:13:54,792 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39315.31 MB 2025-02-14 11:13:54,792 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39315.31 MB 2025-02-14 11:13:54,792 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:13:54,792 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36499.85 MB 2025-02-14 11:13:54,810 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8121, cut from 8123 2025-02-14 11:13:54,810 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 11:13:54,816 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 11:13:54,816 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 11:13:54,816 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:13:54,816 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:13:54,816 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24939.57 MB 2025-02-14 11:13:54,816 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33335.94 MB 2025-02-14 11:13:54,816 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8396.37 MB 2025-02-14 11:13:54,816 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39315.31 MB 2025-02-14 11:13:54,816 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43490.74 MB 2025-02-14 11:13:54,816 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4175.43 MB 2025-02-14 11:13:54,816 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33335.94 MB 2025-02-14 11:13:54,978 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7913] 2025-02-14 11:13:54,980 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:13:54,980 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 11:13:54,981 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:13:54,981 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 11:13:54,986 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 11:13:54,987 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:13:54,987 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 11:13:54,987 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 11:15:09,810 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:15:09,811 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 11:15:09,816 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 11:15:09,820 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:15:09,820 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 196, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 11:15:09,821 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:15:09,821 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 196, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 11:15:12,868 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 11:15:12,868 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 11:15:12,869 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.04 seconds 2025-02-14 11:15:12,869 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:15:12,869 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14334.47 MB 2025-02-14 11:15:12,869 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15028.10 MB 2025-02-14 11:15:12,869 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 693.63 MB 2025-02-14 11:15:12,869 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51837.40 MB 2025-02-14 11:15:12,869 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17381.20 MB 2025-02-14 11:15:12,869 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34456.21 MB 2025-02-14 11:15:12,869 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24033.32 MB 2025-02-14 11:15:12,883 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 11:15:12,883 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 11:15:12,883 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:15:12,883 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:15:12,883 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15028.10 MB 2025-02-14 11:15:12,883 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15237.75 MB 2025-02-14 11:15:12,883 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 209.65 MB 2025-02-14 11:15:12,883 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17381.20 MB 2025-02-14 11:15:12,883 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18643.68 MB 2025-02-14 11:15:12,883 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1262.49 MB 2025-02-14 11:15:12,883 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17543.17 MB 2025-02-14 11:15:13,737 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 11:15:13,737 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 11:15:13,737 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.85 seconds 2025-02-14 11:15:13,737 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:15:13,737 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15237.75 MB 2025-02-14 11:15:13,737 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15473.97 MB 2025-02-14 11:15:13,738 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 236.22 MB 2025-02-14 11:15:13,738 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18643.68 MB 2025-02-14 11:15:13,738 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17540.58 MB 2025-02-14 11:15:13,738 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1103.10 MB 2025-02-14 11:15:13,738 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19409.22 MB 2025-02-14 11:15:13,746 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 11:15:13,746 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 11:15:13,746 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:15:13,746 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:15:13,746 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15473.91 MB 2025-02-14 11:15:13,746 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16314.55 MB 2025-02-14 11:15:13,746 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 840.64 MB 2025-02-14 11:15:13,746 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17540.58 MB 2025-02-14 11:15:13,746 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17962.11 MB 2025-02-14 11:15:13,746 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 421.53 MB 2025-02-14 11:15:13,746 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16945.31 MB 2025-02-14 11:15:13,842 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 11:15:13,842 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 11:15:13,842 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 11:15:13,842 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:15:13,842 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16314.55 MB 2025-02-14 11:15:13,842 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17312.21 MB 2025-02-14 11:15:13,842 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 997.66 MB 2025-02-14 11:15:13,842 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17962.11 MB 2025-02-14 11:15:13,842 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20703.08 MB 2025-02-14 11:15:13,842 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2740.98 MB 2025-02-14 11:15:13,842 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19779.38 MB 2025-02-14 11:15:13,843 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 11:15:13,843 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 11:15:13,843 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 11:15:13,843 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:15:13,843 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15473.91 MB 2025-02-14 11:15:13,843 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17312.21 MB 2025-02-14 11:15:13,843 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1838.30 MB 2025-02-14 11:15:13,843 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17540.58 MB 2025-02-14 11:15:13,843 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20703.08 MB 2025-02-14 11:15:13,843 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3162.51 MB 2025-02-14 11:15:13,843 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19779.38 MB 2025-02-14 11:15:13,919 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 11:15:13,919 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 11:15:13,919 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 11:15:13,919 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:15:13,919 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17994.63 MB 2025-02-14 11:15:13,919 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18335.95 MB 2025-02-14 11:15:13,919 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 341.32 MB 2025-02-14 11:15:13,919 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20703.08 MB 2025-02-14 11:15:13,919 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20885.54 MB 2025-02-14 11:15:13,919 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 182.45 MB 2025-02-14 11:15:13,919 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18657.21 MB 2025-02-14 11:15:13,930 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 11:15:13,930 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 11:15:13,930 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:15:13,930 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:15:13,930 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18519.69 MB 2025-02-14 11:15:13,930 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18723.71 MB 2025-02-14 11:15:13,930 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 204.02 MB 2025-02-14 11:15:13,930 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20885.54 MB 2025-02-14 11:15:13,930 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20889.73 MB 2025-02-14 11:15:13,930 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-14 11:15:13,930 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18754.99 MB 2025-02-14 11:15:13,931 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 11:15:13,931 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 11:15:13,931 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.11 seconds 2025-02-14 11:15:13,931 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:15:13,931 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13651.59 MB 2025-02-14 11:15:13,931 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18924.78 MB 2025-02-14 11:15:13,931 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5273.20 MB 2025-02-14 11:15:13,931 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51837.40 MB 2025-02-14 11:15:13,931 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20889.73 MB 2025-02-14 11:15:13,931 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30947.67 MB 2025-02-14 11:15:13,931 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18924.78 MB 2025-02-14 11:15:14,197 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 11:15:14,197 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 11:15:14,197 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 11:15:14,197 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:15:14,197 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18924.78 MB 2025-02-14 11:15:14,197 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17607.77 MB 2025-02-14 11:15:14,197 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1317.01 MB 2025-02-14 11:15:14,197 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20889.73 MB 2025-02-14 11:15:14,197 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20889.73 MB 2025-02-14 11:15:14,197 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:15:14,197 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19125.75 MB 2025-02-14 11:15:14,215 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 11:15:14,215 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 11:15:14,221 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 11:15:14,221 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 11:15:14,221 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:15:14,221 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:15:14,221 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17607.77 MB 2025-02-14 11:15:14,221 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26046.79 MB 2025-02-14 11:15:14,221 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 11:15:14,221 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20889.73 MB 2025-02-14 11:15:14,221 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31379.69 MB 2025-02-14 11:15:14,221 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 11:15:14,221 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26046.79 MB 2025-02-14 11:15:14,383 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 11:15:14,385 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:15:14,385 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 11:15:14,386 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:15:14,386 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 11:15:14,391 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 11:15:14,392 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:15:14,392 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 11:15:14,392 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 11:15:36,843 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:15:36,844 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 11:15:36,852 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 11:15:36,858 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:15:36,858 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1792, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 11:15:36,860 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:15:36,860 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1792, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 11:16:04,385 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 11:16:04,386 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 11:16:04,386 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.52 seconds 2025-02-14 11:16:04,386 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:16:04,386 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25455.65 MB 2025-02-14 11:16:04,386 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31797.44 MB 2025-02-14 11:16:04,386 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6341.79 MB 2025-02-14 11:16:04,386 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43964.69 MB 2025-02-14 11:16:04,386 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40028.34 MB 2025-02-14 11:16:04,386 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3936.35 MB 2025-02-14 11:16:04,386 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40816.63 MB 2025-02-14 11:16:04,493 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 11:16:04,493 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 11:16:04,493 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 11:16:04,493 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:16:04,493 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31797.44 MB 2025-02-14 11:16:04,493 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25093.90 MB 2025-02-14 11:16:04,493 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6703.54 MB 2025-02-14 11:16:04,493 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40028.34 MB 2025-02-14 11:16:04,493 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 59659.78 MB 2025-02-14 11:16:04,493 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 19631.44 MB 2025-02-14 11:16:04,493 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50443.96 MB 2025-02-14 11:16:06,414 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 11:16:06,414 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 11:16:06,414 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 11:16:06,414 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:16:06,414 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25093.90 MB 2025-02-14 11:16:06,414 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25624.74 MB 2025-02-14 11:16:06,414 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 11:16:06,414 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59659.78 MB 2025-02-14 11:16:06,414 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30907.83 MB 2025-02-14 11:16:06,414 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28751.95 MB 2025-02-14 11:16:06,414 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29604.07 MB 2025-02-14 11:16:06,430 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 11:16:06,430 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 11:16:06,430 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:16:06,430 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:16:06,430 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25624.74 MB 2025-02-14 11:16:06,430 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27514.27 MB 2025-02-14 11:16:06,430 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 11:16:06,430 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30907.83 MB 2025-02-14 11:16:06,430 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31851.54 MB 2025-02-14 11:16:06,430 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 11:16:06,430 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28931.70 MB 2025-02-14 11:16:06,641 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 11:16:06,641 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 11:16:06,641 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 11:16:06,641 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:16:06,641 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27514.27 MB 2025-02-14 11:16:06,641 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29756.13 MB 2025-02-14 11:16:06,641 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 11:16:06,641 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31851.54 MB 2025-02-14 11:16:06,641 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37985.71 MB 2025-02-14 11:16:06,641 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 11:16:06,641 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35300.41 MB 2025-02-14 11:16:06,642 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 11:16:06,642 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 11:16:06,642 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 11:16:06,642 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:16:06,642 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25624.74 MB 2025-02-14 11:16:06,642 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29756.13 MB 2025-02-14 11:16:06,642 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 11:16:06,642 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30907.83 MB 2025-02-14 11:16:06,642 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37985.71 MB 2025-02-14 11:16:06,642 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7077.89 MB 2025-02-14 11:16:06,642 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35300.41 MB 2025-02-14 11:16:06,860 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 11:16:06,860 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 11:16:06,860 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 11:16:06,860 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:16:06,860 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31289.67 MB 2025-02-14 11:16:06,860 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32056.67 MB 2025-02-14 11:16:06,860 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 11:16:06,860 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37985.71 MB 2025-02-14 11:16:06,860 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38403.05 MB 2025-02-14 11:16:06,860 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 11:16:06,860 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32764.46 MB 2025-02-14 11:16:06,879 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 11:16:06,879 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 11:16:06,879 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:16:06,879 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:16:06,879 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32469.56 MB 2025-02-14 11:16:06,879 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32697.74 MB 2025-02-14 11:16:06,879 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.17 MB 2025-02-14 11:16:06,879 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38403.05 MB 2025-02-14 11:16:06,879 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38403.05 MB 2025-02-14 11:16:06,879 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:16:06,880 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32937.30 MB 2025-02-14 11:16:06,881 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 11:16:06,881 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 11:16:06,881 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.02 seconds 2025-02-14 11:16:06,881 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:16:06,881 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19212.18 MB 2025-02-14 11:16:06,881 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32897.83 MB 2025-02-14 11:16:06,881 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13685.65 MB 2025-02-14 11:16:06,881 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43964.69 MB 2025-02-14 11:16:06,881 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38403.05 MB 2025-02-14 11:16:06,881 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5561.65 MB 2025-02-14 11:16:06,881 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32937.30 MB 2025-02-14 11:16:07,147 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 11:16:07,147 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 11:16:07,148 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 11:16:07,148 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:16:07,148 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32897.83 MB 2025-02-14 11:16:07,148 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24201.33 MB 2025-02-14 11:16:07,148 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8696.50 MB 2025-02-14 11:16:07,148 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38403.05 MB 2025-02-14 11:16:07,148 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38403.05 MB 2025-02-14 11:16:07,148 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:16:07,148 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35397.21 MB 2025-02-14 11:16:07,165 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8122, cut from 8124 2025-02-14 11:16:07,166 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 11:16:07,172 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 11:16:07,172 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 11:16:07,172 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:16:07,172 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:16:07,172 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24201.33 MB 2025-02-14 11:16:07,172 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32598.73 MB 2025-02-14 11:16:07,172 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8397.40 MB 2025-02-14 11:16:07,172 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38403.05 MB 2025-02-14 11:16:07,172 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42578.48 MB 2025-02-14 11:16:07,172 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4175.43 MB 2025-02-14 11:16:07,172 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32598.73 MB 2025-02-14 11:16:07,333 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7914] 2025-02-14 11:16:07,335 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:16:07,335 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 11:16:07,336 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:16:07,336 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 11:16:07,340 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 11:16:07,341 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:16:07,341 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 11:16:07,342 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 11:16:17,324 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:16:17,324 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 11:16:17,329 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 11:16:17,333 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:16:17,333 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 442, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 11:16:17,334 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:16:17,334 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 442, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 11:16:24,178 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 11:16:24,178 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 11:16:24,178 - resource_logging.py:150 - __exit__ - DEBUG - Time: 6.84 seconds 2025-02-14 11:16:24,178 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:16:24,178 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16048.63 MB 2025-02-14 11:16:24,178 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17613.11 MB 2025-02-14 11:16:24,178 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1564.48 MB 2025-02-14 11:16:24,178 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50929.34 MB 2025-02-14 11:16:24,178 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21302.87 MB 2025-02-14 11:16:24,178 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29626.47 MB 2025-02-14 11:16:24,178 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26426.78 MB 2025-02-14 11:16:24,206 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 11:16:24,206 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 11:16:24,206 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 11:16:24,206 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:16:24,206 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17613.11 MB 2025-02-14 11:16:24,206 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18020.53 MB 2025-02-14 11:16:24,206 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 407.43 MB 2025-02-14 11:16:24,206 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21302.87 MB 2025-02-14 11:16:24,206 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25425.87 MB 2025-02-14 11:16:24,206 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4123.00 MB 2025-02-14 11:16:24,206 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23156.64 MB 2025-02-14 11:16:26,107 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 11:16:26,107 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 11:16:26,107 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.90 seconds 2025-02-14 11:16:26,107 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:16:26,107 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18020.53 MB 2025-02-14 11:16:26,107 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18540.76 MB 2025-02-14 11:16:26,107 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 520.22 MB 2025-02-14 11:16:26,107 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25425.87 MB 2025-02-14 11:16:26,107 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20856.18 MB 2025-02-14 11:16:26,107 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4569.69 MB 2025-02-14 11:16:26,107 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22531.75 MB 2025-02-14 11:16:26,121 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 11:16:26,121 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 11:16:26,121 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:16:26,121 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:16:26,121 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18540.76 MB 2025-02-14 11:16:26,121 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20392.54 MB 2025-02-14 11:16:26,121 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1851.79 MB 2025-02-14 11:16:26,121 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20856.18 MB 2025-02-14 11:16:26,121 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24094.18 MB 2025-02-14 11:16:26,121 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3238.00 MB 2025-02-14 11:16:26,121 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21782.67 MB 2025-02-14 11:16:26,328 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 11:16:26,328 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 11:16:26,328 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 11:16:26,328 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:16:26,328 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20392.54 MB 2025-02-14 11:16:26,328 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22589.56 MB 2025-02-14 11:16:26,328 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2197.02 MB 2025-02-14 11:16:26,328 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24094.18 MB 2025-02-14 11:16:26,328 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30570.18 MB 2025-02-14 11:16:26,328 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6476.01 MB 2025-02-14 11:16:26,328 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28026.11 MB 2025-02-14 11:16:26,329 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 11:16:26,329 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 11:16:26,329 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 11:16:26,329 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:16:26,329 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18540.76 MB 2025-02-14 11:16:26,329 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22589.56 MB 2025-02-14 11:16:26,329 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4048.81 MB 2025-02-14 11:16:26,329 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20856.18 MB 2025-02-14 11:16:26,329 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30570.18 MB 2025-02-14 11:16:26,329 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9714.01 MB 2025-02-14 11:16:26,329 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28026.11 MB 2025-02-14 11:16:26,491 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 11:16:26,492 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 11:16:26,492 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 11:16:26,492 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:16:26,492 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24093.49 MB 2025-02-14 11:16:26,492 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24847.24 MB 2025-02-14 11:16:26,492 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 753.76 MB 2025-02-14 11:16:26,492 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30570.18 MB 2025-02-14 11:16:26,492 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30979.13 MB 2025-02-14 11:16:26,492 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 408.94 MB 2025-02-14 11:16:26,492 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25540.88 MB 2025-02-14 11:16:26,510 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 11:16:26,510 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 11:16:26,510 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:16:26,510 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:16:26,510 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25251.88 MB 2025-02-14 11:16:26,510 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25483.55 MB 2025-02-14 11:16:26,510 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 231.68 MB 2025-02-14 11:16:26,510 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30979.13 MB 2025-02-14 11:16:26,510 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30979.13 MB 2025-02-14 11:16:26,510 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:16:26,510 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25639.17 MB 2025-02-14 11:16:26,511 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 11:16:26,511 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 11:16:26,511 - resource_logging.py:150 - __exit__ - DEBUG - Time: 9.18 seconds 2025-02-14 11:16:26,511 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:16:26,511 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14508.67 MB 2025-02-14 11:16:26,511 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25684.62 MB 2025-02-14 11:16:26,511 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11175.95 MB 2025-02-14 11:16:26,511 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50929.34 MB 2025-02-14 11:16:26,511 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30979.13 MB 2025-02-14 11:16:26,511 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19950.21 MB 2025-02-14 11:16:26,511 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25684.62 MB 2025-02-14 11:16:26,779 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 11:16:26,779 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 11:16:26,779 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 11:16:26,779 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:16:26,779 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25684.62 MB 2025-02-14 11:16:26,779 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19478.45 MB 2025-02-14 11:16:26,779 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6206.17 MB 2025-02-14 11:16:26,779 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30979.13 MB 2025-02-14 11:16:26,779 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30979.13 MB 2025-02-14 11:16:26,779 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:16:26,779 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28296.76 MB 2025-02-14 11:16:26,797 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 11:16:26,797 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 11:16:26,803 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 11:16:26,803 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 11:16:26,803 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:16:26,803 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:16:26,803 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19478.45 MB 2025-02-14 11:16:26,803 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27917.47 MB 2025-02-14 11:16:26,803 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 11:16:26,803 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30979.13 MB 2025-02-14 11:16:26,803 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41469.08 MB 2025-02-14 11:16:26,803 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 11:16:26,803 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27917.47 MB 2025-02-14 11:16:26,970 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 11:16:26,971 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:16:26,971 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 11:16:26,972 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:16:26,972 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 11:16:26,977 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 11:16:26,978 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:16:26,978 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 11:16:26,978 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 11:16:37,194 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:16:37,194 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 11:16:37,199 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 11:16:37,202 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:16:37,202 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 158, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 11:16:37,203 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:16:37,203 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 158, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 11:16:39,685 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 11:16:39,685 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 11:16:39,685 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.48 seconds 2025-02-14 11:16:39,685 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:16:39,685 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14069.68 MB 2025-02-14 11:16:39,685 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14628.83 MB 2025-02-14 11:16:39,685 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 559.15 MB 2025-02-14 11:16:39,685 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54054.09 MB 2025-02-14 11:16:39,685 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17834.18 MB 2025-02-14 11:16:39,685 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -36219.91 MB 2025-02-14 11:16:39,685 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23541.85 MB 2025-02-14 11:16:39,696 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 11:16:39,696 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 11:16:39,696 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:16:39,696 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:16:39,696 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14628.83 MB 2025-02-14 11:16:39,696 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14639.89 MB 2025-02-14 11:16:39,696 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11.06 MB 2025-02-14 11:16:39,696 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17834.18 MB 2025-02-14 11:16:39,696 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17834.18 MB 2025-02-14 11:16:39,696 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:16:39,696 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16378.01 MB 2025-02-14 11:16:40,282 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 11:16:40,282 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 11:16:40,282 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.58 seconds 2025-02-14 11:16:40,282 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:16:40,282 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14639.89 MB 2025-02-14 11:16:40,282 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14800.47 MB 2025-02-14 11:16:40,282 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 160.58 MB 2025-02-14 11:16:40,282 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17834.18 MB 2025-02-14 11:16:40,282 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17370.71 MB 2025-02-14 11:16:40,282 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -463.47 MB 2025-02-14 11:16:40,282 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18726.43 MB 2025-02-14 11:16:40,289 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 11:16:40,289 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 11:16:40,289 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 11:16:40,289 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:16:40,289 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14800.40 MB 2025-02-14 11:16:40,289 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15371.85 MB 2025-02-14 11:16:40,289 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 571.45 MB 2025-02-14 11:16:40,289 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17370.71 MB 2025-02-14 11:16:40,289 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17370.71 MB 2025-02-14 11:16:40,289 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:16:40,289 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15800.62 MB 2025-02-14 11:16:40,411 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 11:16:40,411 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 11:16:40,411 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 11:16:40,411 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:16:40,411 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15371.85 MB 2025-02-14 11:16:40,411 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16065.91 MB 2025-02-14 11:16:40,411 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 694.07 MB 2025-02-14 11:16:40,411 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17370.71 MB 2025-02-14 11:16:40,411 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18807.26 MB 2025-02-14 11:16:40,411 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1436.55 MB 2025-02-14 11:16:40,411 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17727.15 MB 2025-02-14 11:16:40,412 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 11:16:40,412 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 11:16:40,412 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 11:16:40,412 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:16:40,412 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14800.40 MB 2025-02-14 11:16:40,412 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16065.91 MB 2025-02-14 11:16:40,412 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1265.51 MB 2025-02-14 11:16:40,412 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17370.71 MB 2025-02-14 11:16:40,412 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18807.26 MB 2025-02-14 11:16:40,412 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1436.55 MB 2025-02-14 11:16:40,412 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17727.15 MB 2025-02-14 11:16:40,477 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 11:16:40,477 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 11:16:40,477 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 11:16:40,477 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:16:40,477 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16736.25 MB 2025-02-14 11:16:40,477 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17027.74 MB 2025-02-14 11:16:40,477 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 291.49 MB 2025-02-14 11:16:40,477 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18807.26 MB 2025-02-14 11:16:40,477 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18996.00 MB 2025-02-14 11:16:40,477 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 188.74 MB 2025-02-14 11:16:40,477 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17241.85 MB 2025-02-14 11:16:40,484 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 11:16:40,484 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 11:16:40,484 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 11:16:40,484 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:16:40,484 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17212.12 MB 2025-02-14 11:16:40,484 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17444.69 MB 2025-02-14 11:16:40,484 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 232.57 MB 2025-02-14 11:16:40,484 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18996.00 MB 2025-02-14 11:16:40,484 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18996.00 MB 2025-02-14 11:16:40,484 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:16:40,484 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17444.69 MB 2025-02-14 11:16:40,485 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 11:16:40,485 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 11:16:40,485 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.28 seconds 2025-02-14 11:16:40,485 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:16:40,485 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13519.19 MB 2025-02-14 11:16:40,485 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17645.30 MB 2025-02-14 11:16:40,485 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4126.11 MB 2025-02-14 11:16:40,485 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54054.09 MB 2025-02-14 11:16:40,485 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18996.00 MB 2025-02-14 11:16:40,485 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35058.09 MB 2025-02-14 11:16:40,485 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17645.30 MB 2025-02-14 11:16:40,764 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 11:16:40,764 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 11:16:40,764 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.28 seconds 2025-02-14 11:16:40,764 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:16:40,764 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17645.30 MB 2025-02-14 11:16:40,764 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20652.33 MB 2025-02-14 11:16:40,764 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3007.03 MB 2025-02-14 11:16:40,764 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18996.00 MB 2025-02-14 11:16:40,764 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22485.66 MB 2025-02-14 11:16:40,764 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3489.66 MB 2025-02-14 11:16:40,764 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20953.57 MB 2025-02-14 11:16:40,782 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8143, cut from 8145 2025-02-14 11:16:40,782 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 11:16:40,789 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 11:16:40,789 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 11:16:40,789 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:16:40,789 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:16:40,789 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20652.33 MB 2025-02-14 11:16:40,789 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29071.40 MB 2025-02-14 11:16:40,789 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8419.08 MB 2025-02-14 11:16:40,789 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22485.66 MB 2025-02-14 11:16:40,789 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32950.45 MB 2025-02-14 11:16:40,789 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10464.79 MB 2025-02-14 11:16:40,789 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29071.40 MB 2025-02-14 11:16:40,952 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7935] 2025-02-14 11:16:40,953 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:16:40,953 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 11:16:40,954 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:16:40,954 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 11:16:40,959 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 11:16:40,960 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:16:40,960 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 11:16:40,960 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 11:17:40,596 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:17:40,596 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 11:17:40,604 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 11:17:40,610 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:17:40,611 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 177, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 11:17:40,612 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:17:40,613 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 177, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 11:17:43,387 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 11:17:43,387 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 11:17:43,387 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.77 seconds 2025-02-14 11:17:43,387 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:17:43,387 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14202.07 MB 2025-02-14 11:17:43,387 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14828.46 MB 2025-02-14 11:17:43,387 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 626.39 MB 2025-02-14 11:17:43,387 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41322.28 MB 2025-02-14 11:17:43,387 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17370.71 MB 2025-02-14 11:17:43,387 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23951.57 MB 2025-02-14 11:17:43,387 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23674.51 MB 2025-02-14 11:17:43,406 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 11:17:43,406 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 11:17:43,406 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:17:43,406 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:17:43,406 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14828.46 MB 2025-02-14 11:17:43,406 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15132.61 MB 2025-02-14 11:17:43,406 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 304.14 MB 2025-02-14 11:17:43,406 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17370.71 MB 2025-02-14 11:17:43,406 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18624.81 MB 2025-02-14 11:17:43,406 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1254.10 MB 2025-02-14 11:17:43,406 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17319.73 MB 2025-02-14 11:17:44,260 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 11:17:44,260 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 11:17:44,260 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.85 seconds 2025-02-14 11:17:44,260 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:17:44,260 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15132.61 MB 2025-02-14 11:17:44,260 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15367.50 MB 2025-02-14 11:17:44,260 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 234.90 MB 2025-02-14 11:17:44,260 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18624.81 MB 2025-02-14 11:17:44,260 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18306.04 MB 2025-02-14 11:17:44,260 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -318.77 MB 2025-02-14 11:17:44,260 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19304.08 MB 2025-02-14 11:17:44,269 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 11:17:44,269 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 11:17:44,269 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:17:44,269 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:17:44,269 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15367.44 MB 2025-02-14 11:17:44,269 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16203.35 MB 2025-02-14 11:17:44,269 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 835.92 MB 2025-02-14 11:17:44,269 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18306.04 MB 2025-02-14 11:17:44,269 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18725.47 MB 2025-02-14 11:17:44,269 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 419.43 MB 2025-02-14 11:17:44,269 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16830.57 MB 2025-02-14 11:17:44,368 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 11:17:44,368 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 11:17:44,368 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 11:17:44,368 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:17:44,368 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16203.35 MB 2025-02-14 11:17:44,368 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17195.41 MB 2025-02-14 11:17:44,368 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 992.06 MB 2025-02-14 11:17:44,368 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18725.47 MB 2025-02-14 11:17:44,368 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21242.05 MB 2025-02-14 11:17:44,368 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2516.58 MB 2025-02-14 11:17:44,368 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19648.72 MB 2025-02-14 11:17:44,369 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 11:17:44,369 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 11:17:44,369 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 11:17:44,369 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:17:44,369 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15367.44 MB 2025-02-14 11:17:44,369 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17195.41 MB 2025-02-14 11:17:44,369 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1827.97 MB 2025-02-14 11:17:44,369 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18306.04 MB 2025-02-14 11:17:44,369 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21242.05 MB 2025-02-14 11:17:44,369 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2936.01 MB 2025-02-14 11:17:44,369 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19648.72 MB 2025-02-14 11:17:44,444 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 11:17:44,444 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 11:17:44,444 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 11:17:44,444 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:17:44,444 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17874.00 MB 2025-02-14 11:17:44,444 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18213.40 MB 2025-02-14 11:17:44,444 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 339.40 MB 2025-02-14 11:17:44,444 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21242.05 MB 2025-02-14 11:17:44,444 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21426.60 MB 2025-02-14 11:17:44,444 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 184.55 MB 2025-02-14 11:17:44,444 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18533.29 MB 2025-02-14 11:17:44,454 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 11:17:44,454 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 11:17:44,454 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:17:44,454 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:17:44,454 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18396.11 MB 2025-02-14 11:17:44,454 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18625.15 MB 2025-02-14 11:17:44,454 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.04 MB 2025-02-14 11:17:44,454 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21426.60 MB 2025-02-14 11:17:44,454 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21426.60 MB 2025-02-14 11:17:44,454 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:17:44,454 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18648.36 MB 2025-02-14 11:17:44,455 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 11:17:44,455 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 11:17:44,456 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.84 seconds 2025-02-14 11:17:44,456 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:17:44,456 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13585.39 MB 2025-02-14 11:17:44,456 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18826.22 MB 2025-02-14 11:17:44,456 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5240.84 MB 2025-02-14 11:17:44,456 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41322.28 MB 2025-02-14 11:17:44,456 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21426.60 MB 2025-02-14 11:17:44,456 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19895.68 MB 2025-02-14 11:17:44,456 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18826.22 MB 2025-02-14 11:17:44,719 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 11:17:44,719 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 11:17:44,719 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 11:17:44,719 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:17:44,719 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18826.22 MB 2025-02-14 11:17:44,719 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17537.11 MB 2025-02-14 11:17:44,719 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1289.11 MB 2025-02-14 11:17:44,719 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21426.60 MB 2025-02-14 11:17:44,719 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21426.60 MB 2025-02-14 11:17:44,719 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:17:44,719 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19060.65 MB 2025-02-14 11:17:44,737 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 11:17:44,737 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2,'] 2025-02-14 11:17:44,743 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 11:17:44,743 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 11:17:44,744 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:17:44,744 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:17:44,744 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17537.11 MB 2025-02-14 11:17:44,744 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25976.14 MB 2025-02-14 11:17:44,744 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 11:17:44,744 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21426.60 MB 2025-02-14 11:17:44,744 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31916.56 MB 2025-02-14 11:17:44,744 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 11:17:44,744 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25976.14 MB 2025-02-14 11:17:44,907 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 11:17:44,908 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:17:44,908 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 11:17:44,909 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:17:44,909 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 11:17:44,914 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 11:17:44,915 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:17:44,915 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 11:17:44,915 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2,'] 2025-02-14 11:18:13,838 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:18:13,838 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 11:18:13,843 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 11:18:13,847 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:18:13,847 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1327, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 11:18:13,848 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:18:13,848 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1327, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 11:18:34,125 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 11:18:34,125 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 11:18:34,125 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.27 seconds 2025-02-14 11:18:34,125 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:18:34,125 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22215.45 MB 2025-02-14 11:18:34,125 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26911.63 MB 2025-02-14 11:18:34,125 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4696.18 MB 2025-02-14 11:18:34,125 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44501.57 MB 2025-02-14 11:18:34,125 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38386.27 MB 2025-02-14 11:18:34,125 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6115.30 MB 2025-02-14 11:18:34,125 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35763.69 MB 2025-02-14 11:18:34,203 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 11:18:34,203 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 11:18:34,203 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 11:18:34,203 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:18:34,203 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26911.63 MB 2025-02-14 11:18:34,203 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22676.51 MB 2025-02-14 11:18:34,203 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4235.12 MB 2025-02-14 11:18:34,203 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38386.27 MB 2025-02-14 11:18:34,203 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47659.88 MB 2025-02-14 11:18:34,203 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9273.61 MB 2025-02-14 11:18:34,203 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40908.13 MB 2025-02-14 11:18:36,108 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 11:18:36,108 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 11:18:36,108 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.90 seconds 2025-02-14 11:18:36,108 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:18:36,108 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22676.51 MB 2025-02-14 11:18:36,108 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23207.35 MB 2025-02-14 11:18:36,108 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 11:18:36,108 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47659.88 MB 2025-02-14 11:18:36,108 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33688.65 MB 2025-02-14 11:18:36,108 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13971.23 MB 2025-02-14 11:18:36,108 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27186.68 MB 2025-02-14 11:18:36,121 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 11:18:36,121 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 11:18:36,121 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:18:36,121 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:18:36,121 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23207.35 MB 2025-02-14 11:18:36,121 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25096.89 MB 2025-02-14 11:18:36,121 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 11:18:36,121 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33688.65 MB 2025-02-14 11:18:36,121 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33688.65 MB 2025-02-14 11:18:36,121 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:18:36,121 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26514.31 MB 2025-02-14 11:18:36,335 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 11:18:36,335 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 11:18:36,335 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 11:18:36,335 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:18:36,335 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25096.89 MB 2025-02-14 11:18:36,335 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27338.74 MB 2025-02-14 11:18:36,335 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 11:18:36,335 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33688.65 MB 2025-02-14 11:18:36,335 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36047.95 MB 2025-02-14 11:18:36,335 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2359.30 MB 2025-02-14 11:18:36,335 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32883.02 MB 2025-02-14 11:18:36,336 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 11:18:36,336 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 11:18:36,336 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 11:18:36,336 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:18:36,336 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23207.35 MB 2025-02-14 11:18:36,336 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27338.74 MB 2025-02-14 11:18:36,336 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 11:18:36,336 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33688.65 MB 2025-02-14 11:18:36,336 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36047.95 MB 2025-02-14 11:18:36,336 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2359.30 MB 2025-02-14 11:18:36,336 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32883.02 MB 2025-02-14 11:18:36,503 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 11:18:36,503 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 11:18:36,503 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 11:18:36,503 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:18:36,503 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28872.28 MB 2025-02-14 11:18:36,503 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29639.29 MB 2025-02-14 11:18:36,503 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 11:18:36,503 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36047.95 MB 2025-02-14 11:18:36,503 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36463.18 MB 2025-02-14 11:18:36,503 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 11:18:36,503 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30347.07 MB 2025-02-14 11:18:36,522 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 11:18:36,522 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 11:18:36,522 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:18:36,522 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:18:36,522 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30052.17 MB 2025-02-14 11:18:36,522 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30279.76 MB 2025-02-14 11:18:36,522 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.58 MB 2025-02-14 11:18:36,522 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36463.18 MB 2025-02-14 11:18:36,522 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36463.18 MB 2025-02-14 11:18:36,522 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:18:36,522 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30524.23 MB 2025-02-14 11:18:36,523 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 11:18:36,523 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 11:18:36,523 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.67 seconds 2025-02-14 11:18:36,523 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:18:36,523 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17592.08 MB 2025-02-14 11:18:36,523 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30480.61 MB 2025-02-14 11:18:36,523 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12888.53 MB 2025-02-14 11:18:36,523 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44501.57 MB 2025-02-14 11:18:36,523 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36463.18 MB 2025-02-14 11:18:36,523 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8038.38 MB 2025-02-14 11:18:36,523 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30524.23 MB 2025-02-14 11:18:36,789 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 11:18:36,790 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 11:18:36,790 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 11:18:36,790 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:18:36,790 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30480.61 MB 2025-02-14 11:18:36,790 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22593.04 MB 2025-02-14 11:18:36,790 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7887.57 MB 2025-02-14 11:18:36,790 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36463.18 MB 2025-02-14 11:18:36,790 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36463.18 MB 2025-02-14 11:18:36,790 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:18:36,790 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32989.51 MB 2025-02-14 11:18:36,807 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8153, cut from 8155 2025-02-14 11:18:36,807 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 11:18:36,813 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 11:18:36,813 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 11:18:36,813 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:18:36,813 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:18:36,813 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22593.04 MB 2025-02-14 11:18:36,813 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31023.20 MB 2025-02-14 11:18:36,813 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8430.16 MB 2025-02-14 11:18:36,813 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36463.18 MB 2025-02-14 11:18:36,813 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40653.29 MB 2025-02-14 11:18:36,813 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4190.11 MB 2025-02-14 11:18:36,813 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31023.20 MB 2025-02-14 11:18:36,976 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7945] 2025-02-14 11:18:36,978 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:18:36,978 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 11:18:36,979 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:18:36,979 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 11:18:36,984 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 11:18:36,986 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:18:36,986 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 11:18:36,986 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 11:18:47,681 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:18:47,681 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 11:18:47,686 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 11:18:47,689 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:18:47,689 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 670, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 11:18:47,690 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:18:47,690 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 670, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 11:18:58,081 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 11:18:58,081 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 11:18:58,081 - resource_logging.py:150 - __exit__ - DEBUG - Time: 10.39 seconds 2025-02-14 11:18:58,081 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:18:58,081 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17637.37 MB 2025-02-14 11:18:58,081 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20009.25 MB 2025-02-14 11:18:58,081 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2371.88 MB 2025-02-14 11:18:58,081 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49033.51 MB 2025-02-14 11:18:58,081 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23477.62 MB 2025-02-14 11:18:58,081 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25555.89 MB 2025-02-14 11:18:58,081 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28921.49 MB 2025-02-14 11:18:58,127 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 11:18:58,127 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 11:18:58,127 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-14 11:18:58,127 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:18:58,127 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20009.25 MB 2025-02-14 11:18:58,127 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19260.97 MB 2025-02-14 11:18:58,127 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -748.28 MB 2025-02-14 11:18:58,127 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23477.62 MB 2025-02-14 11:18:58,127 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32459.72 MB 2025-02-14 11:18:58,127 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8982.10 MB 2025-02-14 11:18:58,127 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28948.17 MB 2025-02-14 11:19:00,069 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 11:19:00,069 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 11:19:00,069 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-14 11:19:00,069 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:19:00,069 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19260.97 MB 2025-02-14 11:19:00,069 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19791.81 MB 2025-02-14 11:19:00,069 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 11:19:00,069 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32459.72 MB 2025-02-14 11:19:00,069 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22521.32 MB 2025-02-14 11:19:00,069 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9938.40 MB 2025-02-14 11:19:00,069 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23772.19 MB 2025-02-14 11:19:00,082 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 11:19:00,082 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 11:19:00,082 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:19:00,083 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:19:00,083 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19791.81 MB 2025-02-14 11:19:00,083 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21681.35 MB 2025-02-14 11:19:00,083 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 11:19:00,083 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22521.32 MB 2025-02-14 11:19:00,083 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25352.47 MB 2025-02-14 11:19:00,083 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-14 11:19:00,083 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23098.78 MB 2025-02-14 11:19:00,293 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 11:19:00,293 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 11:19:00,293 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 11:19:00,293 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:19:00,293 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21681.35 MB 2025-02-14 11:19:00,293 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23923.20 MB 2025-02-14 11:19:00,293 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 11:19:00,293 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25352.47 MB 2025-02-14 11:19:00,293 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31486.64 MB 2025-02-14 11:19:00,293 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 11:19:00,293 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29467.49 MB 2025-02-14 11:19:00,294 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 11:19:00,294 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 11:19:00,294 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 11:19:00,294 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:19:00,294 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19791.81 MB 2025-02-14 11:19:00,294 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23923.20 MB 2025-02-14 11:19:00,294 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 11:19:00,294 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22521.32 MB 2025-02-14 11:19:00,294 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31486.64 MB 2025-02-14 11:19:00,294 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8965.32 MB 2025-02-14 11:19:00,294 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29467.49 MB 2025-02-14 11:19:00,606 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 11:19:00,606 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 11:19:00,606 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.31 seconds 2025-02-14 11:19:00,606 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:19:00,606 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25456.75 MB 2025-02-14 11:19:00,606 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26223.75 MB 2025-02-14 11:19:00,606 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 11:19:00,606 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31486.64 MB 2025-02-14 11:19:00,606 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31899.78 MB 2025-02-14 11:19:00,606 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 11:19:00,606 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26931.54 MB 2025-02-14 11:19:00,625 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 11:19:00,625 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 11:19:00,625 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:19:00,625 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:19:00,625 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26636.64 MB 2025-02-14 11:19:00,625 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26865.07 MB 2025-02-14 11:19:00,625 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.43 MB 2025-02-14 11:19:00,625 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31899.78 MB 2025-02-14 11:19:00,625 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31899.78 MB 2025-02-14 11:19:00,625 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:19:00,625 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27086.90 MB 2025-02-14 11:19:00,626 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 11:19:00,626 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 11:19:00,626 - resource_logging.py:150 - __exit__ - DEBUG - Time: 12.93 seconds 2025-02-14 11:19:00,626 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:19:00,627 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15303.04 MB 2025-02-14 11:19:00,627 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27065.33 MB 2025-02-14 11:19:00,627 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11762.29 MB 2025-02-14 11:19:00,627 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49033.51 MB 2025-02-14 11:19:00,627 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31899.78 MB 2025-02-14 11:19:00,627 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17133.73 MB 2025-02-14 11:19:00,627 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27086.90 MB 2025-02-14 11:19:00,893 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 11:19:00,893 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 11:19:00,893 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 11:19:00,893 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:19:00,893 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27065.33 MB 2025-02-14 11:19:00,893 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30067.20 MB 2025-02-14 11:19:00,893 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3001.87 MB 2025-02-14 11:19:00,893 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31899.78 MB 2025-02-14 11:19:00,893 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31899.78 MB 2025-02-14 11:19:00,893 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:19:00,893 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30367.35 MB 2025-02-14 11:19:00,911 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8129, cut from 8131 2025-02-14 11:19:00,911 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-14 11:19:00,917 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 11:19:00,917 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 11:19:00,917 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:19:00,917 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:19:00,917 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30067.20 MB 2025-02-14 11:19:00,917 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38472.31 MB 2025-02-14 11:19:00,917 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8405.11 MB 2025-02-14 11:19:00,917 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31899.78 MB 2025-02-14 11:19:00,917 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42347.79 MB 2025-02-14 11:19:00,917 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10448.01 MB 2025-02-14 11:19:00,917 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38472.31 MB 2025-02-14 11:19:01,082 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7921] 2025-02-14 11:19:01,084 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:19:01,084 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 11:19:01,085 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:19:01,085 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 11:19:01,089 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 11:19:01,090 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:19:01,091 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 11:19:01,091 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-14 11:19:22,586 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:19:22,586 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 11:19:22,591 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 11:19:22,594 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:19:22,594 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 238, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 11:19:22,595 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:19:22,595 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 238, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 11:19:26,278 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 11:19:26,278 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 11:19:26,278 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.68 seconds 2025-02-14 11:19:26,278 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:19:26,278 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32299.51 MB 2025-02-14 11:19:26,278 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33142.57 MB 2025-02-14 11:19:26,278 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 843.06 MB 2025-02-14 11:19:26,278 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54884.56 MB 2025-02-14 11:19:26,278 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35978.74 MB 2025-02-14 11:19:26,278 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18905.83 MB 2025-02-14 11:19:26,279 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41998.43 MB 2025-02-14 11:19:26,294 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 11:19:26,295 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 11:19:26,295 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:19:26,295 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:19:26,295 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33142.57 MB 2025-02-14 11:19:26,295 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33367.72 MB 2025-02-14 11:19:26,295 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 225.15 MB 2025-02-14 11:19:26,295 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35978.74 MB 2025-02-14 11:19:26,295 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37480.30 MB 2025-02-14 11:19:26,295 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1501.56 MB 2025-02-14 11:19:26,295 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36170.92 MB 2025-02-14 11:19:27,319 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 11:19:27,319 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 11:19:27,319 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.02 seconds 2025-02-14 11:19:27,319 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:19:27,319 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33367.72 MB 2025-02-14 11:19:27,319 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33649.07 MB 2025-02-14 11:19:27,319 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 281.35 MB 2025-02-14 11:19:27,319 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37480.30 MB 2025-02-14 11:19:27,319 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35886.47 MB 2025-02-14 11:19:27,319 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1593.84 MB 2025-02-14 11:19:27,319 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37624.13 MB 2025-02-14 11:19:27,328 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 11:19:27,328 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 11:19:27,328 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:19:27,328 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:19:27,328 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33649.07 MB 2025-02-14 11:19:27,328 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34652.37 MB 2025-02-14 11:19:27,328 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1003.31 MB 2025-02-14 11:19:27,328 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35886.47 MB 2025-02-14 11:19:27,328 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36891.00 MB 2025-02-14 11:19:27,328 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1004.54 MB 2025-02-14 11:19:27,328 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35403.61 MB 2025-02-14 11:19:27,441 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 11:19:27,441 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 11:19:27,441 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 11:19:27,442 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:19:27,442 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34652.37 MB 2025-02-14 11:19:27,442 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35841.64 MB 2025-02-14 11:19:27,442 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1189.26 MB 2025-02-14 11:19:27,442 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36891.00 MB 2025-02-14 11:19:27,442 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40149.98 MB 2025-02-14 11:19:27,442 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3258.97 MB 2025-02-14 11:19:27,442 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38784.27 MB 2025-02-14 11:19:27,442 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 11:19:27,442 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 11:19:27,442 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 11:19:27,442 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:19:27,442 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33649.07 MB 2025-02-14 11:19:27,442 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35841.64 MB 2025-02-14 11:19:27,442 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2192.57 MB 2025-02-14 11:19:27,442 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35886.47 MB 2025-02-14 11:19:27,442 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40149.98 MB 2025-02-14 11:19:27,442 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4263.51 MB 2025-02-14 11:19:27,442 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38784.27 MB 2025-02-14 11:19:27,536 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 11:19:27,536 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 11:19:27,536 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 11:19:27,537 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:19:27,537 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36654.41 MB 2025-02-14 11:19:27,537 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19388.54 MB 2025-02-14 11:19:27,537 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -17265.87 MB 2025-02-14 11:19:27,537 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40149.98 MB 2025-02-14 11:19:27,537 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40149.98 MB 2025-02-14 11:19:27,537 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:19:27,537 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36858.72 MB 2025-02-14 11:19:27,548 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 11:19:27,548 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 11:19:27,548 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:19:27,548 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:19:27,548 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19607.38 MB 2025-02-14 11:19:27,548 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19818.35 MB 2025-02-14 11:19:27,548 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 210.97 MB 2025-02-14 11:19:27,548 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40149.98 MB 2025-02-14 11:19:27,548 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40149.98 MB 2025-02-14 11:19:27,548 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:19:27,548 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19834.94 MB 2025-02-14 11:19:27,549 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 11:19:27,549 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 11:19:27,549 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.95 seconds 2025-02-14 11:19:27,549 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:19:27,549 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31470.30 MB 2025-02-14 11:19:27,549 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20019.13 MB 2025-02-14 11:19:27,549 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -11451.18 MB 2025-02-14 11:19:27,549 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54884.56 MB 2025-02-14 11:19:27,549 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40149.98 MB 2025-02-14 11:19:27,549 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14734.59 MB 2025-02-14 11:19:27,549 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20019.13 MB 2025-02-14 11:19:27,816 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 11:19:27,816 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 11:19:27,816 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 11:19:27,816 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:19:27,816 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14900.37 MB 2025-02-14 11:19:27,816 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17909.98 MB 2025-02-14 11:19:27,816 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3009.61 MB 2025-02-14 11:19:27,816 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40149.98 MB 2025-02-14 11:19:27,816 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40149.98 MB 2025-02-14 11:19:27,816 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:19:27,816 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18210.91 MB 2025-02-14 11:19:27,834 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8150, cut from 8152 2025-02-14 11:19:27,834 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 11:19:27,840 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 11:19:27,840 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 11:19:27,840 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:19:27,840 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:19:27,840 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17909.98 MB 2025-02-14 11:19:27,840 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26336.49 MB 2025-02-14 11:19:27,840 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8426.50 MB 2025-02-14 11:19:27,840 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40149.98 MB 2025-02-14 11:19:27,840 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48528.10 MB 2025-02-14 11:19:27,840 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8378.12 MB 2025-02-14 11:19:27,840 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26336.49 MB 2025-02-14 11:19:28,001 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7942] 2025-02-14 11:19:28,002 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:19:28,002 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 11:19:28,003 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:19:28,003 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 11:19:28,008 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 11:19:28,009 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:19:28,009 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 11:19:28,009 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 11:19:38,345 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:19:38,345 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 11:19:38,350 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 11:19:38,354 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:19:38,354 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 406, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 11:19:38,355 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:19:38,355 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 406, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 11:19:44,649 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 11:19:44,649 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 11:19:44,649 - resource_logging.py:150 - __exit__ - DEBUG - Time: 6.29 seconds 2025-02-14 11:19:44,649 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:19:44,649 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15797.78 MB 2025-02-14 11:19:44,649 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17234.59 MB 2025-02-14 11:19:44,649 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1436.81 MB 2025-02-14 11:19:44,649 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61094.23 MB 2025-02-14 11:19:44,649 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20042.48 MB 2025-02-14 11:19:44,649 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -41051.75 MB 2025-02-14 11:19:44,649 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26175.12 MB 2025-02-14 11:19:44,675 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 11:19:44,675 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 11:19:44,675 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:19:44,675 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:19:44,675 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17234.59 MB 2025-02-14 11:19:44,675 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17791.06 MB 2025-02-14 11:19:44,675 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 556.47 MB 2025-02-14 11:19:44,675 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20042.48 MB 2025-02-14 11:19:44,675 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24960.30 MB 2025-02-14 11:19:44,675 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4917.82 MB 2025-02-14 11:19:44,675 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22679.98 MB 2025-02-14 11:19:46,532 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 11:19:46,533 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 11:19:46,533 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.86 seconds 2025-02-14 11:19:46,533 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:19:46,533 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17791.06 MB 2025-02-14 11:19:46,533 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18303.32 MB 2025-02-14 11:19:46,533 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 512.26 MB 2025-02-14 11:19:46,533 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24960.30 MB 2025-02-14 11:19:46,533 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20633.88 MB 2025-02-14 11:19:46,533 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4326.42 MB 2025-02-14 11:19:46,533 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22302.27 MB 2025-02-14 11:19:46,546 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 11:19:46,546 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 11:19:46,546 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:19:46,546 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:19:46,546 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18303.32 MB 2025-02-14 11:19:46,546 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20126.79 MB 2025-02-14 11:19:46,546 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1823.47 MB 2025-02-14 11:19:46,546 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20633.88 MB 2025-02-14 11:19:46,546 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23827.84 MB 2025-02-14 11:19:46,546 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3193.96 MB 2025-02-14 11:19:46,546 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21494.61 MB 2025-02-14 11:19:46,749 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 11:19:46,749 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 11:19:46,749 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 11:19:46,749 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:19:46,750 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20126.79 MB 2025-02-14 11:19:46,750 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22290.19 MB 2025-02-14 11:19:46,750 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2163.39 MB 2025-02-14 11:19:46,750 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23827.84 MB 2025-02-14 11:19:46,750 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29758.59 MB 2025-02-14 11:19:46,750 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5930.75 MB 2025-02-14 11:19:46,750 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27640.42 MB 2025-02-14 11:19:46,750 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 11:19:46,750 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 11:19:46,750 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 11:19:46,750 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:19:46,750 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18303.32 MB 2025-02-14 11:19:46,750 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22290.19 MB 2025-02-14 11:19:46,750 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3986.87 MB 2025-02-14 11:19:46,750 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20633.88 MB 2025-02-14 11:19:46,750 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29758.59 MB 2025-02-14 11:19:46,750 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9124.71 MB 2025-02-14 11:19:46,750 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27640.42 MB 2025-02-14 11:19:46,911 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 11:19:46,911 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 11:19:46,911 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 11:19:46,911 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:19:46,911 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23770.06 MB 2025-02-14 11:19:46,911 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24510.21 MB 2025-02-14 11:19:46,911 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 740.16 MB 2025-02-14 11:19:46,911 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29758.59 MB 2025-02-14 11:19:46,911 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30159.14 MB 2025-02-14 11:19:46,911 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 400.56 MB 2025-02-14 11:19:46,911 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25193.23 MB 2025-02-14 11:19:46,929 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 11:19:46,929 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 11:19:46,929 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:19:46,929 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:19:46,929 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24908.65 MB 2025-02-14 11:19:46,929 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25115.17 MB 2025-02-14 11:19:46,929 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.52 MB 2025-02-14 11:19:46,929 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30159.14 MB 2025-02-14 11:19:46,929 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30163.34 MB 2025-02-14 11:19:46,929 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-14 11:19:46,929 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25304.09 MB 2025-02-14 11:19:46,930 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 11:19:46,930 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 11:19:46,930 - resource_logging.py:150 - __exit__ - DEBUG - Time: 8.57 seconds 2025-02-14 11:19:46,930 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:19:46,930 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14383.24 MB 2025-02-14 11:19:46,930 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25316.24 MB 2025-02-14 11:19:46,931 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 10933.00 MB 2025-02-14 11:19:46,931 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61094.23 MB 2025-02-14 11:19:46,931 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30163.34 MB 2025-02-14 11:19:46,931 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30930.89 MB 2025-02-14 11:19:46,931 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25316.24 MB 2025-02-14 11:19:47,198 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 11:19:47,198 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 11:19:47,198 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 11:19:47,198 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:19:47,198 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25316.24 MB 2025-02-14 11:19:47,199 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19321.04 MB 2025-02-14 11:19:47,199 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5995.20 MB 2025-02-14 11:19:47,199 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30163.34 MB 2025-02-14 11:19:47,199 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30163.34 MB 2025-02-14 11:19:47,199 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:19:47,199 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28028.84 MB 2025-02-14 11:19:47,216 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 11:19:47,217 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 11:19:47,223 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 11:19:47,223 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 11:19:47,223 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:19:47,223 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:19:47,223 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19321.04 MB 2025-02-14 11:19:47,223 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27760.06 MB 2025-02-14 11:19:47,223 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 11:19:47,223 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30163.34 MB 2025-02-14 11:19:47,223 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40653.29 MB 2025-02-14 11:19:47,223 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 11:19:47,223 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27760.06 MB 2025-02-14 11:19:47,385 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 11:19:47,387 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:19:47,387 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 11:19:47,387 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:19:47,388 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 11:19:47,392 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 11:19:47,393 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:19:47,393 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 11:19:47,393 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 11:20:34,211 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:20:34,212 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 11:20:34,217 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 11:20:34,220 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:20:34,220 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 157, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 11:20:34,221 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:20:34,221 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 157, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 11:20:36,651 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 11:20:36,651 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 11:20:36,651 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.43 seconds 2025-02-14 11:20:36,651 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:20:36,651 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14062.71 MB 2025-02-14 11:20:36,651 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14618.32 MB 2025-02-14 11:20:36,651 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 555.61 MB 2025-02-14 11:20:36,651 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53238.30 MB 2025-02-14 11:20:36,651 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17823.69 MB 2025-02-14 11:20:36,651 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35414.61 MB 2025-02-14 11:20:36,651 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23534.89 MB 2025-02-14 11:20:36,663 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 11:20:36,663 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 11:20:36,663 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:20:36,663 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:20:36,663 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14618.32 MB 2025-02-14 11:20:36,663 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14887.65 MB 2025-02-14 11:20:36,663 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 269.32 MB 2025-02-14 11:20:36,663 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17823.69 MB 2025-02-14 11:20:36,663 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18649.97 MB 2025-02-14 11:20:36,663 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 826.28 MB 2025-02-14 11:20:36,663 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16869.75 MB 2025-02-14 11:20:37,422 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 11:20:37,422 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 11:20:37,422 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.76 seconds 2025-02-14 11:20:37,422 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:20:37,422 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14887.65 MB 2025-02-14 11:20:37,422 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15096.28 MB 2025-02-14 11:20:37,422 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 208.63 MB 2025-02-14 11:20:37,422 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18649.97 MB 2025-02-14 11:20:37,422 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18649.97 MB 2025-02-14 11:20:37,422 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:20:37,422 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19060.16 MB 2025-02-14 11:20:37,430 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 11:20:37,430 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 11:20:37,430 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 11:20:37,430 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:20:37,430 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15096.21 MB 2025-02-14 11:20:37,430 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15837.67 MB 2025-02-14 11:20:37,430 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 741.46 MB 2025-02-14 11:20:37,430 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18649.97 MB 2025-02-14 11:20:37,430 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18649.97 MB 2025-02-14 11:20:37,430 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:20:37,430 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16394.02 MB 2025-02-14 11:20:37,516 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 11:20:37,516 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 11:20:37,516 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 11:20:37,516 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:20:37,516 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15837.67 MB 2025-02-14 11:20:37,516 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16718.52 MB 2025-02-14 11:20:37,516 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 880.85 MB 2025-02-14 11:20:37,516 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18649.97 MB 2025-02-14 11:20:37,516 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19765.66 MB 2025-02-14 11:20:37,516 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1115.68 MB 2025-02-14 11:20:37,516 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18896.19 MB 2025-02-14 11:20:37,517 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 11:20:37,517 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 11:20:37,517 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 11:20:37,517 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:20:37,517 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15096.21 MB 2025-02-14 11:20:37,517 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16718.52 MB 2025-02-14 11:20:37,517 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1622.31 MB 2025-02-14 11:20:37,517 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18649.97 MB 2025-02-14 11:20:37,517 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19765.66 MB 2025-02-14 11:20:37,517 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1115.68 MB 2025-02-14 11:20:37,517 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18896.19 MB 2025-02-14 11:20:37,585 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 11:20:37,585 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 11:20:37,585 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 11:20:37,585 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:20:37,585 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17320.44 MB 2025-02-14 11:20:37,585 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17621.49 MB 2025-02-14 11:20:37,585 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 301.05 MB 2025-02-14 11:20:37,585 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19765.66 MB 2025-02-14 11:20:37,585 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19925.04 MB 2025-02-14 11:20:37,585 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 159.38 MB 2025-02-14 11:20:37,585 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17907.34 MB 2025-02-14 11:20:37,594 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 11:20:37,594 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 11:20:37,594 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:20:37,594 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:20:37,594 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17783.56 MB 2025-02-14 11:20:37,594 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18000.38 MB 2025-02-14 11:20:37,594 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 216.82 MB 2025-02-14 11:20:37,594 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19925.04 MB 2025-02-14 11:20:37,594 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19925.04 MB 2025-02-14 11:20:37,594 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:20:37,594 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18021.92 MB 2025-02-14 11:20:37,596 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 11:20:37,596 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 11:20:37,596 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.37 seconds 2025-02-14 11:20:37,596 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:20:37,596 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13515.71 MB 2025-02-14 11:20:37,596 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18201.18 MB 2025-02-14 11:20:37,596 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4685.48 MB 2025-02-14 11:20:37,596 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53238.30 MB 2025-02-14 11:20:37,596 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19925.04 MB 2025-02-14 11:20:37,596 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33313.26 MB 2025-02-14 11:20:37,596 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18201.18 MB 2025-02-14 11:20:37,861 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 11:20:37,861 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 11:20:37,861 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 11:20:37,861 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:20:37,861 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18201.18 MB 2025-02-14 11:20:37,861 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17369.12 MB 2025-02-14 11:20:37,861 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -832.07 MB 2025-02-14 11:20:37,861 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19925.04 MB 2025-02-14 11:20:37,861 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19925.04 MB 2025-02-14 11:20:37,861 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:20:37,861 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19003.83 MB 2025-02-14 11:20:37,879 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8151, cut from 8153 2025-02-14 11:20:37,879 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 11:20:37,885 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 11:20:37,885 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 11:20:37,885 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:20:37,885 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:20:37,885 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17369.12 MB 2025-02-14 11:20:37,885 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25796.45 MB 2025-02-14 11:20:37,885 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8427.34 MB 2025-02-14 11:20:37,885 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19925.04 MB 2025-02-14 11:20:37,885 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30400.32 MB 2025-02-14 11:20:37,885 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10475.27 MB 2025-02-14 11:20:37,885 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25796.45 MB 2025-02-14 11:20:38,051 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7943] 2025-02-14 11:20:38,052 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:20:38,052 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 11:20:38,053 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:20:38,053 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 11:20:38,060 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 11:20:38,061 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:20:38,061 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 11:20:38,061 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 11:21:01,614 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:21:01,614 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 11:21:01,619 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 11:21:01,623 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:21:01,623 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1217, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 11:21:01,624 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:21:01,624 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1217, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 11:21:20,369 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 11:21:20,369 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 11:21:20,369 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.74 seconds 2025-02-14 11:21:20,369 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:21:20,369 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21448.96 MB 2025-02-14 11:21:20,369 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25756.51 MB 2025-02-14 11:21:20,369 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4307.55 MB 2025-02-14 11:21:20,369 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38780.53 MB 2025-02-14 11:21:20,369 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35882.27 MB 2025-02-14 11:21:20,369 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2898.26 MB 2025-02-14 11:21:20,369 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34770.70 MB 2025-02-14 11:21:20,440 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 11:21:20,440 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 11:21:20,440 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 11:21:20,440 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:21:20,440 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25756.51 MB 2025-02-14 11:21:20,440 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22104.65 MB 2025-02-14 11:21:20,440 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3651.85 MB 2025-02-14 11:21:20,440 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35882.27 MB 2025-02-14 11:21:20,440 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44432.36 MB 2025-02-14 11:21:20,440 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8550.09 MB 2025-02-14 11:21:20,440 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38614.98 MB 2025-02-14 11:21:22,350 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 11:21:22,350 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 11:21:22,350 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 11:21:22,350 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:21:22,350 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22104.65 MB 2025-02-14 11:21:22,350 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22635.50 MB 2025-02-14 11:21:22,350 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 11:21:22,350 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44432.36 MB 2025-02-14 11:21:22,350 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31574.72 MB 2025-02-14 11:21:22,350 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12857.64 MB 2025-02-14 11:21:22,350 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26614.83 MB 2025-02-14 11:21:22,363 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 11:21:22,363 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 11:21:22,363 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:21:22,363 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:21:22,363 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22635.50 MB 2025-02-14 11:21:22,363 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24525.03 MB 2025-02-14 11:21:22,363 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 11:21:22,363 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31574.72 MB 2025-02-14 11:21:22,363 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31574.72 MB 2025-02-14 11:21:22,363 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:21:22,363 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25942.46 MB 2025-02-14 11:21:22,576 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 11:21:22,576 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 11:21:22,576 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 11:21:22,576 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:21:22,576 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24525.03 MB 2025-02-14 11:21:22,576 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26766.89 MB 2025-02-14 11:21:22,576 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 11:21:22,576 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31574.72 MB 2025-02-14 11:21:22,576 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34877.73 MB 2025-02-14 11:21:22,576 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 11:21:22,576 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32311.17 MB 2025-02-14 11:21:22,577 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 11:21:22,577 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 11:21:22,577 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 11:21:22,577 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:21:22,577 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22635.50 MB 2025-02-14 11:21:22,577 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26766.89 MB 2025-02-14 11:21:22,577 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 11:21:22,577 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31574.72 MB 2025-02-14 11:21:22,577 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34877.73 MB 2025-02-14 11:21:22,577 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 11:21:22,577 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32311.17 MB 2025-02-14 11:21:22,747 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 11:21:22,747 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 11:21:22,747 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 11:21:22,747 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:21:22,747 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28300.43 MB 2025-02-14 11:21:22,747 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29067.43 MB 2025-02-14 11:21:22,747 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 11:21:22,747 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34877.73 MB 2025-02-14 11:21:22,747 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35292.97 MB 2025-02-14 11:21:22,747 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 11:21:22,747 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29775.22 MB 2025-02-14 11:21:22,768 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 11:21:22,768 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 11:21:22,768 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:21:22,768 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:21:22,768 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29480.32 MB 2025-02-14 11:21:22,768 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29708.41 MB 2025-02-14 11:21:22,768 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.09 MB 2025-02-14 11:21:22,768 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35292.97 MB 2025-02-14 11:21:22,768 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35292.97 MB 2025-02-14 11:21:22,768 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:21:22,768 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29954.44 MB 2025-02-14 11:21:22,769 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 11:21:22,769 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 11:21:22,769 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.14 seconds 2025-02-14 11:21:22,769 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:21:22,769 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17208.83 MB 2025-02-14 11:21:22,769 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29908.55 MB 2025-02-14 11:21:22,769 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12699.71 MB 2025-02-14 11:21:22,769 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38780.53 MB 2025-02-14 11:21:22,769 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35292.97 MB 2025-02-14 11:21:22,769 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3487.56 MB 2025-02-14 11:21:22,769 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29954.44 MB 2025-02-14 11:21:23,038 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 11:21:23,038 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 11:21:23,038 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 11:21:23,038 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:21:23,038 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29908.55 MB 2025-02-14 11:21:23,038 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22198.75 MB 2025-02-14 11:21:23,038 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7709.80 MB 2025-02-14 11:21:23,038 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35292.97 MB 2025-02-14 11:21:23,038 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35292.97 MB 2025-02-14 11:21:23,038 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:21:23,038 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32408.54 MB 2025-02-14 11:21:23,055 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8124, cut from 8126 2025-02-14 11:21:23,056 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2.'] 2025-02-14 11:21:23,062 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 11:21:23,062 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 11:21:23,062 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:21:23,062 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:21:23,062 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22198.75 MB 2025-02-14 11:21:23,062 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30599.17 MB 2025-02-14 11:21:23,062 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8400.42 MB 2025-02-14 11:21:23,062 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35292.97 MB 2025-02-14 11:21:23,062 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39468.40 MB 2025-02-14 11:21:23,062 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4175.43 MB 2025-02-14 11:21:23,062 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30599.17 MB 2025-02-14 11:21:23,224 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7916] 2025-02-14 11:21:23,225 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:21:23,225 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 11:21:23,226 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:21:23,226 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 11:21:23,231 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 11:21:23,232 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:21:23,232 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 11:21:23,232 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2.'] 2025-02-14 11:22:52,119 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:22:52,119 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 11:22:52,124 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 11:22:52,129 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:22:52,129 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 409, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 11:22:52,130 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:22:52,130 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 409, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 11:22:58,440 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 11:22:58,441 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 11:22:58,441 - resource_logging.py:150 - __exit__ - DEBUG - Time: 6.31 seconds 2025-02-14 11:22:58,441 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:22:58,441 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15818.68 MB 2025-02-14 11:22:58,441 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17266.11 MB 2025-02-14 11:22:58,441 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1447.43 MB 2025-02-14 11:22:58,441 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47819.26 MB 2025-02-14 11:22:58,441 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20243.81 MB 2025-02-14 11:22:58,441 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27575.45 MB 2025-02-14 11:22:58,441 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26196.02 MB 2025-02-14 11:22:58,467 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 11:22:58,467 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 11:22:58,467 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:22:58,467 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:22:58,467 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17266.11 MB 2025-02-14 11:22:58,467 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17806.65 MB 2025-02-14 11:22:58,467 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 540.54 MB 2025-02-14 11:22:58,467 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20243.81 MB 2025-02-14 11:22:58,467 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24987.57 MB 2025-02-14 11:22:58,467 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4743.76 MB 2025-02-14 11:22:58,467 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22722.11 MB 2025-02-14 11:23:00,329 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 11:23:00,329 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 11:23:00,329 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.86 seconds 2025-02-14 11:23:00,329 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:23:00,329 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17806.65 MB 2025-02-14 11:23:00,329 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18318.92 MB 2025-02-14 11:23:00,329 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 512.26 MB 2025-02-14 11:23:00,329 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24987.57 MB 2025-02-14 11:23:00,329 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20824.72 MB 2025-02-14 11:23:00,329 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4162.85 MB 2025-02-14 11:23:00,329 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22317.87 MB 2025-02-14 11:23:00,342 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 11:23:00,343 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 11:23:00,343 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:23:00,343 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:23:00,343 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18318.92 MB 2025-02-14 11:23:00,343 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20142.39 MB 2025-02-14 11:23:00,343 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1823.47 MB 2025-02-14 11:23:00,343 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20824.72 MB 2025-02-14 11:23:00,343 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24018.68 MB 2025-02-14 11:23:00,343 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3193.96 MB 2025-02-14 11:23:00,343 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21510.21 MB 2025-02-14 11:23:00,553 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 11:23:00,553 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 11:23:00,553 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 11:23:00,553 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:23:00,553 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20142.39 MB 2025-02-14 11:23:00,553 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22305.78 MB 2025-02-14 11:23:00,553 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2163.39 MB 2025-02-14 11:23:00,553 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24018.68 MB 2025-02-14 11:23:00,553 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29949.43 MB 2025-02-14 11:23:00,553 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5930.75 MB 2025-02-14 11:23:00,553 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27656.01 MB 2025-02-14 11:23:00,554 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 11:23:00,554 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 11:23:00,554 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 11:23:00,554 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:23:00,554 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18318.92 MB 2025-02-14 11:23:00,554 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22305.78 MB 2025-02-14 11:23:00,554 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3986.87 MB 2025-02-14 11:23:00,554 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20824.72 MB 2025-02-14 11:23:00,554 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29949.43 MB 2025-02-14 11:23:00,554 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9124.71 MB 2025-02-14 11:23:00,554 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27656.01 MB 2025-02-14 11:23:00,724 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 11:23:00,724 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 11:23:00,724 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 11:23:00,724 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:23:00,724 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23785.65 MB 2025-02-14 11:23:00,724 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24525.81 MB 2025-02-14 11:23:00,724 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 740.16 MB 2025-02-14 11:23:00,724 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29949.43 MB 2025-02-14 11:23:00,724 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30352.08 MB 2025-02-14 11:23:00,725 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 402.65 MB 2025-02-14 11:23:00,725 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25208.83 MB 2025-02-14 11:23:00,744 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 11:23:00,744 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 11:23:00,744 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:23:00,744 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:23:00,744 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24924.25 MB 2025-02-14 11:23:00,744 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25130.63 MB 2025-02-14 11:23:00,744 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.38 MB 2025-02-14 11:23:00,744 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30352.08 MB 2025-02-14 11:23:00,744 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30356.28 MB 2025-02-14 11:23:00,744 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-14 11:23:00,744 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25299.71 MB 2025-02-14 11:23:00,745 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 11:23:00,745 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 11:23:00,745 - resource_logging.py:150 - __exit__ - DEBUG - Time: 8.61 seconds 2025-02-14 11:23:00,745 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:23:00,745 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14393.70 MB 2025-02-14 11:23:00,745 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25331.70 MB 2025-02-14 11:23:00,745 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 10938.01 MB 2025-02-14 11:23:00,745 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47819.26 MB 2025-02-14 11:23:00,745 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30356.28 MB 2025-02-14 11:23:00,745 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17462.98 MB 2025-02-14 11:23:00,745 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25331.70 MB 2025-02-14 11:23:01,013 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 11:23:01,014 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 11:23:01,014 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 11:23:01,014 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:23:01,014 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25331.70 MB 2025-02-14 11:23:01,014 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19331.49 MB 2025-02-14 11:23:01,014 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6000.21 MB 2025-02-14 11:23:01,014 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30356.28 MB 2025-02-14 11:23:01,014 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30356.28 MB 2025-02-14 11:23:01,014 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:23:01,014 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28044.31 MB 2025-02-14 11:23:01,032 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 11:23:01,032 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 11:23:01,038 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 11:23:01,038 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 11:23:01,038 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:23:01,038 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:23:01,038 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19331.49 MB 2025-02-14 11:23:01,038 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27770.51 MB 2025-02-14 11:23:01,038 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 11:23:01,038 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30356.28 MB 2025-02-14 11:23:01,038 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40846.23 MB 2025-02-14 11:23:01,038 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 11:23:01,038 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27770.51 MB 2025-02-14 11:23:01,210 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 11:23:01,212 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:23:01,212 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 11:23:01,213 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:23:01,213 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 11:23:01,218 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 11:23:01,219 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:23:01,219 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 11:23:01,219 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 11:23:57,813 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:23:57,814 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 11:23:57,819 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 11:23:57,824 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:23:57,824 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2109, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 11:23:57,825 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:23:57,825 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2109, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 11:24:30,367 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 11:24:30,367 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 11:24:30,367 - resource_logging.py:150 - __exit__ - DEBUG - Time: 32.53 seconds 2025-02-14 11:24:30,367 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:24:30,367 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27664.56 MB 2025-02-14 11:24:30,367 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35128.32 MB 2025-02-14 11:24:30,367 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7463.76 MB 2025-02-14 11:24:30,367 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53431.24 MB 2025-02-14 11:24:30,367 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41152.41 MB 2025-02-14 11:24:30,367 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12278.82 MB 2025-02-14 11:24:30,367 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43931.51 MB 2025-02-14 11:24:30,496 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 11:24:30,496 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 11:24:30,496 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 11:24:30,496 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:24:30,497 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35128.32 MB 2025-02-14 11:24:30,497 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26741.88 MB 2025-02-14 11:24:30,497 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8386.44 MB 2025-02-14 11:24:30,497 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41152.41 MB 2025-02-14 11:24:30,497 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 64047.02 MB 2025-02-14 11:24:30,497 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 22894.61 MB 2025-02-14 11:24:30,497 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54543.79 MB 2025-02-14 11:24:32,438 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 11:24:32,438 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 11:24:32,438 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-14 11:24:32,438 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:24:32,438 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26741.88 MB 2025-02-14 11:24:32,438 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27272.73 MB 2025-02-14 11:24:32,438 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 11:24:32,438 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64047.02 MB 2025-02-14 11:24:32,438 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30909.92 MB 2025-02-14 11:24:32,438 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33137.10 MB 2025-02-14 11:24:32,438 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31253.10 MB 2025-02-14 11:24:32,452 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 11:24:32,452 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 11:24:32,452 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:24:32,452 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:24:32,452 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27272.73 MB 2025-02-14 11:24:32,452 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29162.26 MB 2025-02-14 11:24:32,452 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 11:24:32,452 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30909.92 MB 2025-02-14 11:24:32,452 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33741.08 MB 2025-02-14 11:24:32,452 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-14 11:24:32,452 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30579.69 MB 2025-02-14 11:24:32,664 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 11:24:32,665 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 11:24:32,665 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 11:24:32,665 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:24:32,665 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29162.26 MB 2025-02-14 11:24:32,665 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31404.12 MB 2025-02-14 11:24:32,665 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 11:24:32,665 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33741.08 MB 2025-02-14 11:24:32,665 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39403.39 MB 2025-02-14 11:24:32,665 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 11:24:32,665 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36948.40 MB 2025-02-14 11:24:32,665 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 11:24:32,665 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 11:24:32,665 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 11:24:32,665 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:24:32,665 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27272.73 MB 2025-02-14 11:24:32,665 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31404.12 MB 2025-02-14 11:24:32,665 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 11:24:32,665 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30909.92 MB 2025-02-14 11:24:32,665 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39403.39 MB 2025-02-14 11:24:32,665 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8493.47 MB 2025-02-14 11:24:32,665 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36948.40 MB 2025-02-14 11:24:32,841 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 11:24:32,841 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 11:24:32,841 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 11:24:32,841 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:24:32,841 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32937.66 MB 2025-02-14 11:24:32,841 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33704.66 MB 2025-02-14 11:24:32,841 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 11:24:32,841 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39403.39 MB 2025-02-14 11:24:32,841 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39818.63 MB 2025-02-14 11:24:32,841 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 11:24:32,841 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34412.45 MB 2025-02-14 11:24:32,860 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 11:24:32,860 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 11:24:32,860 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:24:32,860 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:24:32,860 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34117.55 MB 2025-02-14 11:24:32,860 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34346.17 MB 2025-02-14 11:24:32,860 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.62 MB 2025-02-14 11:24:32,860 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39818.63 MB 2025-02-14 11:24:32,860 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39818.63 MB 2025-02-14 11:24:32,860 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:24:32,860 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34568.49 MB 2025-02-14 11:24:32,861 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 11:24:32,861 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 11:24:32,861 - resource_logging.py:150 - __exit__ - DEBUG - Time: 35.03 seconds 2025-02-14 11:24:32,861 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:24:32,861 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20316.63 MB 2025-02-14 11:24:32,861 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34546.70 MB 2025-02-14 11:24:32,861 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14230.07 MB 2025-02-14 11:24:32,861 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53431.24 MB 2025-02-14 11:24:32,861 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39818.63 MB 2025-02-14 11:24:32,861 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13612.61 MB 2025-02-14 11:24:32,861 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34568.49 MB 2025-02-14 11:24:33,131 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 11:24:33,131 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 11:24:33,131 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 11:24:33,131 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:24:33,131 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34546.70 MB 2025-02-14 11:24:33,131 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25312.64 MB 2025-02-14 11:24:33,131 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9234.06 MB 2025-02-14 11:24:33,131 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39818.63 MB 2025-02-14 11:24:33,131 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39818.63 MB 2025-02-14 11:24:33,131 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:24:33,131 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37051.61 MB 2025-02-14 11:24:33,149 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8140, cut from 8142 2025-02-14 11:24:33,149 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 11:24:33,155 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 11:24:33,155 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 11:24:33,156 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:24:33,156 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:24:33,156 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25312.64 MB 2025-02-14 11:24:33,156 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33729.24 MB 2025-02-14 11:24:33,156 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8416.60 MB 2025-02-14 11:24:33,156 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39818.63 MB 2025-02-14 11:24:33,156 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48186.26 MB 2025-02-14 11:24:33,156 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8367.64 MB 2025-02-14 11:24:33,156 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33729.24 MB 2025-02-14 11:24:33,322 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7932] 2025-02-14 11:24:33,323 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:24:33,323 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 11:24:33,324 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:24:33,324 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 11:24:33,329 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 11:24:33,330 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:24:33,330 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 11:24:33,330 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 11:25:14,613 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:25:14,613 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 11:25:14,619 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 11:25:14,623 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:25:14,623 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1099, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 11:25:14,624 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:25:14,624 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1099, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 11:25:31,555 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 11:25:31,555 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 11:25:31,555 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.92 seconds 2025-02-14 11:25:31,555 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:25:31,555 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20626.71 MB 2025-02-14 11:25:31,555 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24516.93 MB 2025-02-14 11:25:31,555 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3890.22 MB 2025-02-14 11:25:31,555 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56553.90 MB 2025-02-14 11:25:31,555 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29167.19 MB 2025-02-14 11:25:31,555 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27386.71 MB 2025-02-14 11:25:31,555 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33496.28 MB 2025-02-14 11:25:31,630 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 11:25:31,630 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 11:25:31,630 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 11:25:31,630 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:25:31,630 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24516.93 MB 2025-02-14 11:25:31,630 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21491.21 MB 2025-02-14 11:25:31,630 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3025.72 MB 2025-02-14 11:25:31,630 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29167.19 MB 2025-02-14 11:25:31,630 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43146.81 MB 2025-02-14 11:25:31,630 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 13979.62 MB 2025-02-14 11:25:31,630 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36500.85 MB 2025-02-14 11:25:33,535 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 11:25:33,535 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 11:25:33,535 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.90 seconds 2025-02-14 11:25:33,535 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:25:33,535 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21491.21 MB 2025-02-14 11:25:33,535 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22022.05 MB 2025-02-14 11:25:33,535 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 11:25:33,535 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43146.81 MB 2025-02-14 11:25:33,535 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26692.55 MB 2025-02-14 11:25:33,535 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16454.25 MB 2025-02-14 11:25:33,535 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26001.38 MB 2025-02-14 11:25:33,549 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 11:25:33,549 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 11:25:33,549 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:25:33,549 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:25:33,549 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22022.05 MB 2025-02-14 11:25:33,549 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23911.58 MB 2025-02-14 11:25:33,549 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 11:25:33,549 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26692.55 MB 2025-02-14 11:25:33,549 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28579.99 MB 2025-02-14 11:25:33,549 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 11:25:33,549 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25329.01 MB 2025-02-14 11:25:33,771 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 11:25:33,771 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 11:25:33,771 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 11:25:33,771 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:25:33,771 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23911.58 MB 2025-02-14 11:25:33,771 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26154.36 MB 2025-02-14 11:25:33,772 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2242.78 MB 2025-02-14 11:25:33,772 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28579.99 MB 2025-02-14 11:25:33,772 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34242.30 MB 2025-02-14 11:25:33,772 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 11:25:33,772 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31698.64 MB 2025-02-14 11:25:33,772 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 11:25:33,772 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 11:25:33,772 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.24 seconds 2025-02-14 11:25:33,772 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:25:33,772 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22022.05 MB 2025-02-14 11:25:33,772 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26154.36 MB 2025-02-14 11:25:33,772 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4132.31 MB 2025-02-14 11:25:33,772 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26692.55 MB 2025-02-14 11:25:33,772 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34242.30 MB 2025-02-14 11:25:33,772 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-14 11:25:33,772 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31698.64 MB 2025-02-14 11:25:33,944 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 11:25:33,944 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 11:25:33,944 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 11:25:33,944 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:25:33,944 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27687.90 MB 2025-02-14 11:25:33,944 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28454.90 MB 2025-02-14 11:25:33,944 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 11:25:33,944 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34242.30 MB 2025-02-14 11:25:33,944 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34657.53 MB 2025-02-14 11:25:33,944 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 11:25:33,944 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29162.69 MB 2025-02-14 11:25:33,963 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 11:25:33,963 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 11:25:33,963 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:25:33,963 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:25:33,963 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28867.79 MB 2025-02-14 11:25:33,963 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29096.38 MB 2025-02-14 11:25:33,963 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.58 MB 2025-02-14 11:25:33,963 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34657.53 MB 2025-02-14 11:25:33,963 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34657.53 MB 2025-02-14 11:25:33,963 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:25:33,963 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29335.35 MB 2025-02-14 11:25:33,964 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 11:25:33,964 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 11:25:33,964 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.34 seconds 2025-02-14 11:25:33,964 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:25:33,964 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16797.71 MB 2025-02-14 11:25:33,964 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29296.52 MB 2025-02-14 11:25:33,964 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12498.81 MB 2025-02-14 11:25:33,964 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56553.90 MB 2025-02-14 11:25:33,964 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34657.53 MB 2025-02-14 11:25:33,964 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21896.36 MB 2025-02-14 11:25:33,964 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29335.35 MB 2025-02-14 11:25:34,230 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 11:25:34,231 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 11:25:34,231 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 11:25:34,231 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:25:34,231 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29296.52 MB 2025-02-14 11:25:34,231 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21787.62 MB 2025-02-14 11:25:34,231 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7508.89 MB 2025-02-14 11:25:34,231 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34657.53 MB 2025-02-14 11:25:34,231 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34657.53 MB 2025-02-14 11:25:34,231 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:25:34,231 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31796.51 MB 2025-02-14 11:25:34,249 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8124, cut from 8126 2025-02-14 11:25:34,249 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 11:25:34,255 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 11:25:34,255 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 11:25:34,255 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:25:34,255 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:25:34,255 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21787.62 MB 2025-02-14 11:25:34,255 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30188.49 MB 2025-02-14 11:25:34,255 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8400.86 MB 2025-02-14 11:25:34,255 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34657.53 MB 2025-02-14 11:25:34,255 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43008.39 MB 2025-02-14 11:25:34,255 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8350.86 MB 2025-02-14 11:25:34,255 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30188.49 MB 2025-02-14 11:25:34,417 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7916] 2025-02-14 11:25:34,418 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:25:34,418 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 11:25:34,419 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:25:34,419 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 11:25:34,424 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 11:25:34,425 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:25:34,425 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 11:25:34,425 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 11:26:41,571 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:26:41,571 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 11:26:41,580 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 11:26:41,588 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:26:41,588 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 882, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 11:26:41,590 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:26:41,590 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 882, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 11:26:55,195 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 11:26:55,195 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 11:26:55,195 - resource_logging.py:150 - __exit__ - DEBUG - Time: 13.60 seconds 2025-02-14 11:26:55,195 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:26:55,195 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19114.62 MB 2025-02-14 11:26:55,195 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22235.97 MB 2025-02-14 11:26:55,195 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3121.35 MB 2025-02-14 11:26:55,195 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51359.25 MB 2025-02-14 11:26:55,195 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28391.24 MB 2025-02-14 11:26:55,195 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22968.01 MB 2025-02-14 11:26:55,195 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31078.22 MB 2025-02-14 11:26:55,251 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 11:26:55,251 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 11:26:55,251 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-14 11:26:55,251 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:26:55,251 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22235.97 MB 2025-02-14 11:26:55,251 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20363.09 MB 2025-02-14 11:26:55,251 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1872.88 MB 2025-02-14 11:26:55,251 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28391.24 MB 2025-02-14 11:26:55,251 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37480.30 MB 2025-02-14 11:26:55,251 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9089.06 MB 2025-02-14 11:26:55,251 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32550.43 MB 2025-02-14 11:26:57,211 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 11:26:57,211 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 11:26:57,211 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.96 seconds 2025-02-14 11:26:57,211 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:26:57,211 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20363.09 MB 2025-02-14 11:26:57,211 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20893.94 MB 2025-02-14 11:26:57,211 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 11:26:57,211 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37480.30 MB 2025-02-14 11:26:57,211 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26684.16 MB 2025-02-14 11:26:57,211 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10796.14 MB 2025-02-14 11:26:57,211 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24873.27 MB 2025-02-14 11:26:57,225 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 11:26:57,226 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 11:26:57,226 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:26:57,226 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:26:57,226 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20893.94 MB 2025-02-14 11:26:57,226 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22783.47 MB 2025-02-14 11:26:57,226 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 11:26:57,226 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26684.16 MB 2025-02-14 11:26:57,226 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27627.88 MB 2025-02-14 11:26:57,226 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 11:26:57,226 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24200.90 MB 2025-02-14 11:26:57,440 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 11:26:57,440 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 11:26:57,440 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 11:26:57,440 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:26:57,440 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22783.47 MB 2025-02-14 11:26:57,440 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25025.33 MB 2025-02-14 11:26:57,440 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 11:26:57,440 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27627.88 MB 2025-02-14 11:26:57,440 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33290.19 MB 2025-02-14 11:26:57,440 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 11:26:57,440 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30569.61 MB 2025-02-14 11:26:57,441 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 11:26:57,441 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 11:26:57,441 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 11:26:57,441 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:26:57,441 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20893.94 MB 2025-02-14 11:26:57,441 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25025.33 MB 2025-02-14 11:26:57,441 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 11:26:57,441 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26684.16 MB 2025-02-14 11:26:57,441 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33290.19 MB 2025-02-14 11:26:57,441 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 11:26:57,441 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30569.61 MB 2025-02-14 11:26:57,612 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 11:26:57,612 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 11:26:57,613 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 11:26:57,613 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:26:57,613 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26558.87 MB 2025-02-14 11:26:57,613 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27325.87 MB 2025-02-14 11:26:57,613 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 11:26:57,613 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33290.19 MB 2025-02-14 11:26:57,613 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33705.43 MB 2025-02-14 11:26:57,613 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 11:26:57,613 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28033.66 MB 2025-02-14 11:26:57,632 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 11:26:57,632 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 11:26:57,632 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:26:57,632 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:26:57,632 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27738.76 MB 2025-02-14 11:26:57,632 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27965.95 MB 2025-02-14 11:26:57,632 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.19 MB 2025-02-14 11:26:57,632 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33705.43 MB 2025-02-14 11:26:57,632 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33705.43 MB 2025-02-14 11:26:57,632 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:26:57,632 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28152.95 MB 2025-02-14 11:26:57,633 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 11:26:57,633 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 11:26:57,633 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.04 seconds 2025-02-14 11:26:57,633 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:26:57,633 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16041.67 MB 2025-02-14 11:26:57,633 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28166.58 MB 2025-02-14 11:26:57,633 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12124.92 MB 2025-02-14 11:26:57,633 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51359.25 MB 2025-02-14 11:26:57,633 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33705.43 MB 2025-02-14 11:26:57,633 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17653.83 MB 2025-02-14 11:26:57,633 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28166.58 MB 2025-02-14 11:26:57,902 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 11:26:57,902 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 11:26:57,902 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 11:26:57,902 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:26:57,902 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28166.58 MB 2025-02-14 11:26:57,902 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21039.20 MB 2025-02-14 11:26:57,902 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7127.38 MB 2025-02-14 11:26:57,902 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33705.43 MB 2025-02-14 11:26:57,902 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33705.43 MB 2025-02-14 11:26:57,902 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:26:57,902 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30672.72 MB 2025-02-14 11:26:57,920 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8144, cut from 8146 2025-02-14 11:26:57,920 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 11:26:57,927 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 11:26:57,927 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 11:26:57,927 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:26:57,927 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:26:57,927 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21039.20 MB 2025-02-14 11:26:57,927 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29459.97 MB 2025-02-14 11:26:57,927 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8420.78 MB 2025-02-14 11:26:57,927 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33705.43 MB 2025-02-14 11:26:57,927 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42077.26 MB 2025-02-14 11:26:57,927 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8371.83 MB 2025-02-14 11:26:57,927 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29459.97 MB 2025-02-14 11:26:58,090 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7936] 2025-02-14 11:26:58,092 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:26:58,092 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 11:26:58,093 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:26:58,093 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 11:26:58,097 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 11:26:58,098 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:26:58,098 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 11:26:58,099 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 11:28:31,404 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:28:31,404 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 11:28:31,409 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 11:28:31,413 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:28:31,413 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1413, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 11:28:31,414 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:28:31,414 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1413, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 11:28:53,004 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 11:28:53,004 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 11:28:53,004 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.58 seconds 2025-02-14 11:28:53,004 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:28:53,004 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22814.72 MB 2025-02-14 11:28:53,004 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27815.24 MB 2025-02-14 11:28:53,004 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5000.53 MB 2025-02-14 11:28:53,004 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50449.09 MB 2025-02-14 11:28:53,004 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38644.22 MB 2025-02-14 11:28:53,004 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11804.87 MB 2025-02-14 11:28:53,004 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36815.94 MB 2025-02-14 11:28:53,087 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 11:28:53,087 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 11:28:53,087 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 11:28:53,087 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:28:53,087 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27815.24 MB 2025-02-14 11:28:53,087 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23123.60 MB 2025-02-14 11:28:53,087 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4691.65 MB 2025-02-14 11:28:53,087 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38644.22 MB 2025-02-14 11:28:53,087 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48540.68 MB 2025-02-14 11:28:53,087 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9896.46 MB 2025-02-14 11:28:53,087 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42716.69 MB 2025-02-14 11:28:54,990 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 11:28:54,990 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 11:28:54,990 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.90 seconds 2025-02-14 11:28:54,990 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:28:54,990 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23123.60 MB 2025-02-14 11:28:54,991 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23654.44 MB 2025-02-14 11:28:54,991 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 11:28:54,991 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48540.68 MB 2025-02-14 11:28:54,991 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33642.51 MB 2025-02-14 11:28:54,991 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14898.17 MB 2025-02-14 11:28:54,991 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27633.05 MB 2025-02-14 11:28:55,004 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 11:28:55,004 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 11:28:55,004 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:28:55,004 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:28:55,004 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23654.44 MB 2025-02-14 11:28:55,004 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25543.97 MB 2025-02-14 11:28:55,004 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 11:28:55,004 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33642.51 MB 2025-02-14 11:28:55,004 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33642.51 MB 2025-02-14 11:28:55,004 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:28:55,004 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26961.40 MB 2025-02-14 11:28:55,218 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 11:28:55,218 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 11:28:55,218 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 11:28:55,218 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:28:55,218 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25543.97 MB 2025-02-14 11:28:55,218 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27785.83 MB 2025-02-14 11:28:55,218 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 11:28:55,218 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33642.51 MB 2025-02-14 11:28:55,218 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37417.39 MB 2025-02-14 11:28:55,219 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 11:28:55,219 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33330.11 MB 2025-02-14 11:28:55,219 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 11:28:55,219 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 11:28:55,219 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 11:28:55,219 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:28:55,219 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23654.44 MB 2025-02-14 11:28:55,219 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27785.83 MB 2025-02-14 11:28:55,219 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 11:28:55,219 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33642.51 MB 2025-02-14 11:28:55,219 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37417.39 MB 2025-02-14 11:28:55,219 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 11:28:55,219 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33330.11 MB 2025-02-14 11:28:55,393 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 11:28:55,393 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 11:28:55,393 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 11:28:55,393 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:28:55,393 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29319.37 MB 2025-02-14 11:28:55,393 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30086.37 MB 2025-02-14 11:28:55,393 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 11:28:55,393 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37417.39 MB 2025-02-14 11:28:55,393 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37830.52 MB 2025-02-14 11:28:55,393 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 11:28:55,393 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30794.16 MB 2025-02-14 11:28:55,412 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 11:28:55,412 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 11:28:55,412 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:28:55,413 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:28:55,413 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30499.26 MB 2025-02-14 11:28:55,413 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30728.37 MB 2025-02-14 11:28:55,413 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.11 MB 2025-02-14 11:28:55,413 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37830.52 MB 2025-02-14 11:28:55,413 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37830.52 MB 2025-02-14 11:28:55,413 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:28:55,413 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30970.11 MB 2025-02-14 11:28:55,414 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 11:28:55,414 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 11:28:55,414 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.00 seconds 2025-02-14 11:28:55,414 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:28:55,414 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17891.71 MB 2025-02-14 11:28:55,414 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30929.39 MB 2025-02-14 11:28:55,414 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13037.68 MB 2025-02-14 11:28:55,414 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50449.09 MB 2025-02-14 11:28:55,414 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37830.52 MB 2025-02-14 11:28:55,414 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12618.56 MB 2025-02-14 11:28:55,414 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30970.11 MB 2025-02-14 11:28:55,682 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 11:28:55,682 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 11:28:55,682 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 11:28:55,682 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:28:55,683 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30929.39 MB 2025-02-14 11:28:55,683 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22895.34 MB 2025-02-14 11:28:55,683 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8034.06 MB 2025-02-14 11:28:55,683 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37830.52 MB 2025-02-14 11:28:55,683 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37830.52 MB 2025-02-14 11:28:55,683 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:28:55,683 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33440.45 MB 2025-02-14 11:28:55,700 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8160, cut from 8162 2025-02-14 11:28:55,701 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 11:28:55,707 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 11:28:55,707 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 11:28:55,707 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:28:55,707 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:28:55,707 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22895.34 MB 2025-02-14 11:28:55,707 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31332.81 MB 2025-02-14 11:28:55,707 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8437.47 MB 2025-02-14 11:28:55,707 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37830.52 MB 2025-02-14 11:28:55,707 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46219.13 MB 2025-02-14 11:28:55,707 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8388.61 MB 2025-02-14 11:28:55,707 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31332.81 MB 2025-02-14 11:28:55,870 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7952] 2025-02-14 11:28:55,871 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:28:55,871 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 11:28:55,872 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:28:55,872 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 11:28:55,877 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 11:28:55,878 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:28:55,878 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 11:28:55,878 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 11:30:20,929 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:30:20,929 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 11:30:20,934 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 11:30:20,938 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:30:20,938 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2077, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 11:30:20,939 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:30:20,939 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2077, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 11:30:52,746 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 11:30:52,746 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 11:30:52,746 - resource_logging.py:150 - __exit__ - DEBUG - Time: 31.80 seconds 2025-02-14 11:30:52,746 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:30:52,746 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27441.57 MB 2025-02-14 11:30:52,746 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34792.09 MB 2025-02-14 11:30:52,746 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7350.52 MB 2025-02-14 11:30:52,746 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54607.74 MB 2025-02-14 11:30:52,746 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40997.22 MB 2025-02-14 11:30:52,746 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13610.52 MB 2025-02-14 11:30:52,746 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43709.30 MB 2025-02-14 11:30:52,877 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 11:30:52,877 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 11:30:52,877 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 11:30:52,877 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:30:52,877 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34792.09 MB 2025-02-14 11:30:52,877 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26575.53 MB 2025-02-14 11:30:52,877 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8216.57 MB 2025-02-14 11:30:52,877 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40997.22 MB 2025-02-14 11:30:52,877 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 66028.83 MB 2025-02-14 11:30:52,877 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 25031.61 MB 2025-02-14 11:30:52,877 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 55899.22 MB 2025-02-14 11:30:54,823 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 11:30:54,823 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 11:30:54,823 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-14 11:30:54,823 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:30:54,823 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26575.53 MB 2025-02-14 11:30:54,823 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27106.37 MB 2025-02-14 11:30:54,823 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 11:30:54,823 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 66028.83 MB 2025-02-14 11:30:54,823 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30886.85 MB 2025-02-14 11:30:54,823 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35141.98 MB 2025-02-14 11:30:54,823 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31086.74 MB 2025-02-14 11:30:54,837 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 11:30:54,837 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 11:30:54,837 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:30:54,837 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:30:54,837 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27106.37 MB 2025-02-14 11:30:54,837 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28995.90 MB 2025-02-14 11:30:54,837 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 11:30:54,837 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30886.85 MB 2025-02-14 11:30:54,837 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33718.01 MB 2025-02-14 11:30:54,837 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-14 11:30:54,837 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30413.33 MB 2025-02-14 11:30:55,047 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 11:30:55,047 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 11:30:55,047 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 11:30:55,047 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:30:55,047 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28995.90 MB 2025-02-14 11:30:55,047 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31237.76 MB 2025-02-14 11:30:55,047 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 11:30:55,047 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33718.01 MB 2025-02-14 11:30:55,047 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39380.32 MB 2025-02-14 11:30:55,047 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 11:30:55,047 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36782.04 MB 2025-02-14 11:30:55,048 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 11:30:55,048 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 11:30:55,048 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 11:30:55,048 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:30:55,048 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27106.37 MB 2025-02-14 11:30:55,048 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31237.76 MB 2025-02-14 11:30:55,048 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 11:30:55,048 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30886.85 MB 2025-02-14 11:30:55,048 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39380.32 MB 2025-02-14 11:30:55,048 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8493.47 MB 2025-02-14 11:30:55,048 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36782.04 MB 2025-02-14 11:30:55,221 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 11:30:55,221 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 11:30:55,221 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 11:30:55,221 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:30:55,221 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32771.30 MB 2025-02-14 11:30:55,221 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33538.30 MB 2025-02-14 11:30:55,221 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 11:30:55,221 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39380.32 MB 2025-02-14 11:30:55,221 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39797.65 MB 2025-02-14 11:30:55,221 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 11:30:55,221 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34246.09 MB 2025-02-14 11:30:55,240 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 11:30:55,240 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 11:30:55,240 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:30:55,240 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:30:55,240 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33951.19 MB 2025-02-14 11:30:55,240 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34180.00 MB 2025-02-14 11:30:55,240 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.81 MB 2025-02-14 11:30:55,240 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39797.65 MB 2025-02-14 11:30:55,240 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39797.65 MB 2025-02-14 11:30:55,240 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:30:55,240 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34401.94 MB 2025-02-14 11:30:55,242 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 11:30:55,242 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 11:30:55,242 - resource_logging.py:150 - __exit__ - DEBUG - Time: 34.30 seconds 2025-02-14 11:30:55,242 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:30:55,242 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20205.14 MB 2025-02-14 11:30:55,242 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34380.73 MB 2025-02-14 11:30:55,242 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14175.59 MB 2025-02-14 11:30:55,242 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54607.74 MB 2025-02-14 11:30:55,242 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39797.65 MB 2025-02-14 11:30:55,242 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14810.09 MB 2025-02-14 11:30:55,242 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34401.94 MB 2025-02-14 11:30:55,510 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 11:30:55,510 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 11:30:55,510 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 11:30:55,510 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:30:55,510 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34380.73 MB 2025-02-14 11:30:55,510 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25204.20 MB 2025-02-14 11:30:55,510 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9176.54 MB 2025-02-14 11:30:55,510 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39797.65 MB 2025-02-14 11:30:55,510 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39797.65 MB 2025-02-14 11:30:55,510 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:30:55,510 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36888.10 MB 2025-02-14 11:30:55,528 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8148, cut from 8150 2025-02-14 11:30:55,528 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 11:30:55,534 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 11:30:55,534 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 11:30:55,534 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:30:55,534 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:30:55,534 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25204.20 MB 2025-02-14 11:30:55,534 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33629.15 MB 2025-02-14 11:30:55,534 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8424.95 MB 2025-02-14 11:30:55,534 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39797.65 MB 2025-02-14 11:30:55,534 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48173.68 MB 2025-02-14 11:30:55,534 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8376.03 MB 2025-02-14 11:30:55,534 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33629.15 MB 2025-02-14 11:30:55,697 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7940] 2025-02-14 11:30:55,698 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:30:55,698 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 11:30:55,699 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:30:55,699 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 11:30:55,704 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 11:30:55,705 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:30:55,705 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 11:30:55,705 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 11:33:11,617 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:33:11,617 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 11:33:11,622 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 11:33:11,626 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:33:11,626 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2041, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 11:33:11,627 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:33:11,627 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2041, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 11:33:42,861 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 11:33:42,861 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 11:33:42,861 - resource_logging.py:150 - __exit__ - DEBUG - Time: 31.22 seconds 2025-02-14 11:33:42,861 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:33:42,861 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27190.72 MB 2025-02-14 11:33:42,861 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34413.71 MB 2025-02-14 11:33:42,861 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7222.98 MB 2025-02-14 11:33:42,861 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56549.70 MB 2025-02-14 11:33:42,861 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40879.78 MB 2025-02-14 11:33:42,861 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15669.92 MB 2025-02-14 11:33:42,861 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43231.18 MB 2025-02-14 11:33:42,988 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 11:33:42,988 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 11:33:42,988 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 11:33:42,988 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:33:42,988 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34413.71 MB 2025-02-14 11:33:42,988 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26388.37 MB 2025-02-14 11:33:42,988 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8025.33 MB 2025-02-14 11:33:42,988 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40879.78 MB 2025-02-14 11:33:42,988 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 62251.86 MB 2025-02-14 11:33:42,988 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 21372.08 MB 2025-02-14 11:33:42,988 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 53012.25 MB 2025-02-14 11:33:44,955 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 11:33:44,955 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 11:33:44,955 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.96 seconds 2025-02-14 11:33:44,955 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:33:44,955 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26388.37 MB 2025-02-14 11:33:44,955 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26919.22 MB 2025-02-14 11:33:44,955 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 11:33:44,955 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62251.86 MB 2025-02-14 11:33:44,955 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30886.85 MB 2025-02-14 11:33:44,955 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31365.01 MB 2025-02-14 11:33:44,955 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30899.59 MB 2025-02-14 11:33:44,969 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 11:33:44,969 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 11:33:44,969 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:33:44,969 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:33:44,969 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26919.22 MB 2025-02-14 11:33:44,969 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28808.75 MB 2025-02-14 11:33:44,969 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 11:33:44,969 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30886.85 MB 2025-02-14 11:33:44,969 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32774.29 MB 2025-02-14 11:33:44,969 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 11:33:44,969 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30226.18 MB 2025-02-14 11:33:45,178 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 11:33:45,178 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 11:33:45,178 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 11:33:45,178 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:33:45,178 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28808.75 MB 2025-02-14 11:33:45,178 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31050.61 MB 2025-02-14 11:33:45,178 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 11:33:45,178 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32774.29 MB 2025-02-14 11:33:45,178 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38908.46 MB 2025-02-14 11:33:45,178 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 11:33:45,178 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36594.89 MB 2025-02-14 11:33:45,178 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 11:33:45,178 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 11:33:45,178 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 11:33:45,178 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:33:45,178 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26919.22 MB 2025-02-14 11:33:45,178 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31050.61 MB 2025-02-14 11:33:45,179 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 11:33:45,179 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30886.85 MB 2025-02-14 11:33:45,179 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38908.46 MB 2025-02-14 11:33:45,179 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-14 11:33:45,179 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36594.89 MB 2025-02-14 11:33:45,345 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 11:33:45,345 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 11:33:45,345 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 11:33:45,345 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:33:45,346 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32584.15 MB 2025-02-14 11:33:45,346 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33351.15 MB 2025-02-14 11:33:45,346 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 11:33:45,346 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38908.46 MB 2025-02-14 11:33:45,346 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39323.70 MB 2025-02-14 11:33:45,346 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 11:33:45,346 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34058.94 MB 2025-02-14 11:33:45,364 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 11:33:45,364 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 11:33:45,364 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:33:45,365 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:33:45,365 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33764.04 MB 2025-02-14 11:33:45,365 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33993.02 MB 2025-02-14 11:33:45,365 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.99 MB 2025-02-14 11:33:45,365 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39323.70 MB 2025-02-14 11:33:45,365 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39323.70 MB 2025-02-14 11:33:45,365 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:33:45,365 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34212.05 MB 2025-02-14 11:33:45,366 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 11:33:45,366 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 11:33:45,366 - resource_logging.py:150 - __exit__ - DEBUG - Time: 33.74 seconds 2025-02-14 11:33:45,366 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:33:45,366 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20079.71 MB 2025-02-14 11:33:45,366 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34193.75 MB 2025-02-14 11:33:45,366 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14114.04 MB 2025-02-14 11:33:45,366 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56549.70 MB 2025-02-14 11:33:45,366 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39323.70 MB 2025-02-14 11:33:45,366 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17226.01 MB 2025-02-14 11:33:45,366 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34212.05 MB 2025-02-14 11:33:45,633 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 11:33:45,633 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 11:33:45,633 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 11:33:45,633 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:33:45,633 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34193.75 MB 2025-02-14 11:33:45,633 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25078.77 MB 2025-02-14 11:33:45,633 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9114.98 MB 2025-02-14 11:33:45,633 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39323.70 MB 2025-02-14 11:33:45,633 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39323.70 MB 2025-02-14 11:33:45,633 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:33:45,633 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36701.12 MB 2025-02-14 11:33:45,651 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8148, cut from 8150 2025-02-14 11:33:45,651 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 11:33:45,657 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 11:33:45,657 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 11:33:45,657 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:33:45,657 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:33:45,657 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25078.77 MB 2025-02-14 11:33:45,657 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33503.72 MB 2025-02-14 11:33:45,657 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8424.95 MB 2025-02-14 11:33:45,657 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39323.70 MB 2025-02-14 11:33:45,657 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43511.71 MB 2025-02-14 11:33:45,657 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4188.01 MB 2025-02-14 11:33:45,657 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33503.72 MB 2025-02-14 11:33:45,809 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7940] 2025-02-14 11:33:45,810 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:33:45,810 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 11:33:45,811 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:33:45,811 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 11:33:45,815 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 11:33:45,816 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:33:45,817 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 11:33:45,817 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 11:34:37,994 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:34:37,994 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 11:34:38,000 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 11:34:38,005 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:34:38,005 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2715, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 11:34:38,007 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:34:38,007 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2715, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 11:35:20,200 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 11:35:20,200 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 11:35:20,200 - resource_logging.py:150 - __exit__ - DEBUG - Time: 42.18 seconds 2025-02-14 11:35:20,200 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:35:20,200 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31889.21 MB 2025-02-14 11:35:20,200 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41498.36 MB 2025-02-14 11:35:20,200 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 9609.15 MB 2025-02-14 11:35:20,200 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 70808.24 MB 2025-02-14 11:35:20,200 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45438.99 MB 2025-02-14 11:35:20,200 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25369.25 MB 2025-02-14 11:35:20,200 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51106.59 MB 2025-02-14 11:35:20,378 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 11:35:20,378 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 11:35:20,378 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-14 11:35:20,378 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:35:20,378 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41498.36 MB 2025-02-14 11:35:20,378 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29893.78 MB 2025-02-14 11:35:20,378 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -11604.58 MB 2025-02-14 11:35:20,378 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45438.99 MB 2025-02-14 11:35:20,378 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 78865.50 MB 2025-02-14 11:35:20,378 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 33426.51 MB 2025-02-14 11:35:20,378 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 67097.50 MB 2025-02-14 11:35:22,344 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 11:35:22,344 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 11:35:22,344 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.96 seconds 2025-02-14 11:35:22,344 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:35:22,344 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29893.78 MB 2025-02-14 11:35:22,344 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30424.62 MB 2025-02-14 11:35:22,344 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 11:35:22,344 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 78865.50 MB 2025-02-14 11:35:22,344 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32440.84 MB 2025-02-14 11:35:22,344 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -46424.65 MB 2025-02-14 11:35:22,344 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34404.99 MB 2025-02-14 11:35:22,358 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 11:35:22,358 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 11:35:22,358 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:35:22,358 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:35:22,358 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30424.62 MB 2025-02-14 11:35:22,358 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32314.16 MB 2025-02-14 11:35:22,358 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 11:35:22,358 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32440.84 MB 2025-02-14 11:35:22,358 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35743.86 MB 2025-02-14 11:35:22,358 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 11:35:22,358 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33731.59 MB 2025-02-14 11:35:22,570 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 11:35:22,570 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 11:35:22,570 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 11:35:22,570 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:35:22,570 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32314.16 MB 2025-02-14 11:35:22,570 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34556.01 MB 2025-02-14 11:35:22,570 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 11:35:22,570 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35743.86 MB 2025-02-14 11:35:22,570 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42349.89 MB 2025-02-14 11:35:22,570 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 11:35:22,570 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40100.29 MB 2025-02-14 11:35:22,571 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 11:35:22,571 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 11:35:22,571 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 11:35:22,571 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:35:22,571 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30424.62 MB 2025-02-14 11:35:22,571 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34556.01 MB 2025-02-14 11:35:22,571 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 11:35:22,571 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32440.84 MB 2025-02-14 11:35:22,571 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42349.89 MB 2025-02-14 11:35:22,571 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9909.04 MB 2025-02-14 11:35:22,571 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40100.29 MB 2025-02-14 11:35:22,741 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 11:35:22,741 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 11:35:22,741 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 11:35:22,741 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:35:22,741 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36089.55 MB 2025-02-14 11:35:22,741 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36856.56 MB 2025-02-14 11:35:22,741 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 11:35:22,741 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42349.89 MB 2025-02-14 11:35:22,741 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42765.12 MB 2025-02-14 11:35:22,741 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 11:35:22,741 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37564.34 MB 2025-02-14 11:35:22,760 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 11:35:22,760 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 11:35:22,760 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:35:22,760 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:35:22,760 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37269.45 MB 2025-02-14 11:35:22,760 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37498.14 MB 2025-02-14 11:35:22,760 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.69 MB 2025-02-14 11:35:22,760 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42765.12 MB 2025-02-14 11:35:22,760 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42765.12 MB 2025-02-14 11:35:22,760 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:35:22,760 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37731.22 MB 2025-02-14 11:35:22,761 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 11:35:22,761 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 11:35:22,761 - resource_logging.py:150 - __exit__ - DEBUG - Time: 44.75 seconds 2025-02-14 11:35:22,762 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:35:22,762 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22428.96 MB 2025-02-14 11:35:22,762 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37698.74 MB 2025-02-14 11:35:22,762 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 15269.78 MB 2025-02-14 11:35:22,762 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61347.99 MB 2025-02-14 11:35:22,762 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42765.12 MB 2025-02-14 11:35:22,762 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18582.86 MB 2025-02-14 11:35:22,762 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37731.22 MB 2025-02-14 11:35:23,033 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 11:35:23,033 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 11:35:23,033 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 11:35:23,033 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:35:23,033 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37698.74 MB 2025-02-14 11:35:23,033 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27426.11 MB 2025-02-14 11:35:23,033 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10272.63 MB 2025-02-14 11:35:23,033 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42765.12 MB 2025-02-14 11:35:23,033 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42765.12 MB 2025-02-14 11:35:23,033 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:35:23,033 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40204.57 MB 2025-02-14 11:35:23,052 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8143, cut from 8145 2025-02-14 11:35:23,052 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 11:35:23,058 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 11:35:23,058 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 11:35:23,058 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:35:23,058 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:35:23,058 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27426.11 MB 2025-02-14 11:35:23,058 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35845.19 MB 2025-02-14 11:35:23,058 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8419.08 MB 2025-02-14 11:35:23,058 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42765.12 MB 2025-02-14 11:35:23,058 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46951.04 MB 2025-02-14 11:35:23,058 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4185.92 MB 2025-02-14 11:35:23,058 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35845.19 MB 2025-02-14 11:35:23,221 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7935] 2025-02-14 11:35:23,223 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:35:23,223 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 11:35:23,224 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:35:23,224 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 11:35:23,228 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 11:35:23,229 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:35:23,230 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 11:35:23,230 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 11:36:23,791 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:36:23,791 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 11:36:23,799 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 11:36:23,806 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:36:23,806 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1257, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 11:36:23,808 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:36:23,808 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1257, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 11:36:43,242 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 11:36:43,242 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 11:36:43,242 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.42 seconds 2025-02-14 11:36:43,242 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:36:43,242 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21727.68 MB 2025-02-14 11:36:43,242 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26176.14 MB 2025-02-14 11:36:43,242 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4448.45 MB 2025-02-14 11:36:43,242 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55322.87 MB 2025-02-14 11:36:43,242 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38570.82 MB 2025-02-14 11:36:43,242 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16752.05 MB 2025-02-14 11:36:43,242 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35049.43 MB 2025-02-14 11:36:43,318 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 11:36:43,318 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 11:36:43,319 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 11:36:43,319 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:36:43,319 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26176.14 MB 2025-02-14 11:36:43,319 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22312.60 MB 2025-02-14 11:36:43,319 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3863.53 MB 2025-02-14 11:36:43,319 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38570.82 MB 2025-02-14 11:36:43,319 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45176.85 MB 2025-02-14 11:36:43,319 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 11:36:43,319 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39485.03 MB 2025-02-14 11:36:45,244 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 11:36:45,244 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 11:36:45,244 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 11:36:45,244 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:36:45,244 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22312.60 MB 2025-02-14 11:36:45,244 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22843.44 MB 2025-02-14 11:36:45,244 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 11:36:45,244 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45176.85 MB 2025-02-14 11:36:45,244 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34120.66 MB 2025-02-14 11:36:45,244 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11056.19 MB 2025-02-14 11:36:45,244 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26822.78 MB 2025-02-14 11:36:45,258 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 11:36:45,258 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 11:36:45,258 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:36:45,258 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:36:45,258 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22843.44 MB 2025-02-14 11:36:45,258 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24732.98 MB 2025-02-14 11:36:45,258 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 11:36:45,258 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34120.66 MB 2025-02-14 11:36:45,258 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34120.66 MB 2025-02-14 11:36:45,258 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:36:45,258 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26150.41 MB 2025-02-14 11:36:45,476 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 11:36:45,476 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 11:36:45,476 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 11:36:45,476 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:36:45,476 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24732.98 MB 2025-02-14 11:36:45,476 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26974.83 MB 2025-02-14 11:36:45,476 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 11:36:45,476 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34120.66 MB 2025-02-14 11:36:45,476 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35064.38 MB 2025-02-14 11:36:45,476 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 11:36:45,476 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32519.11 MB 2025-02-14 11:36:45,477 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 11:36:45,477 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 11:36:45,477 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 11:36:45,477 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:36:45,477 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22843.44 MB 2025-02-14 11:36:45,477 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26974.83 MB 2025-02-14 11:36:45,477 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 11:36:45,477 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34120.66 MB 2025-02-14 11:36:45,477 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35064.38 MB 2025-02-14 11:36:45,477 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 11:36:45,477 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32519.11 MB 2025-02-14 11:36:45,651 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 11:36:45,651 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 11:36:45,651 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 11:36:45,651 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:36:45,651 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28508.38 MB 2025-02-14 11:36:45,651 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29275.38 MB 2025-02-14 11:36:45,651 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 11:36:45,651 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35064.38 MB 2025-02-14 11:36:45,651 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35481.71 MB 2025-02-14 11:36:45,651 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 11:36:45,651 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29983.17 MB 2025-02-14 11:36:45,670 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 11:36:45,670 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 11:36:45,670 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:36:45,670 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:36:45,670 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29688.27 MB 2025-02-14 11:36:45,670 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29916.88 MB 2025-02-14 11:36:45,670 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.62 MB 2025-02-14 11:36:45,670 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35481.71 MB 2025-02-14 11:36:45,670 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35481.71 MB 2025-02-14 11:36:45,670 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:36:45,670 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30149.43 MB 2025-02-14 11:36:45,671 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 11:36:45,671 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 11:36:45,672 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.86 seconds 2025-02-14 11:36:45,672 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:36:45,672 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17348.19 MB 2025-02-14 11:36:45,672 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30117.41 MB 2025-02-14 11:36:45,672 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12769.22 MB 2025-02-14 11:36:45,672 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55322.87 MB 2025-02-14 11:36:45,672 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35481.71 MB 2025-02-14 11:36:45,672 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19841.16 MB 2025-02-14 11:36:45,672 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30149.43 MB 2025-02-14 11:36:45,943 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 11:36:45,943 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 11:36:45,943 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 11:36:45,943 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:36:45,943 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30117.41 MB 2025-02-14 11:36:45,943 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22344.20 MB 2025-02-14 11:36:45,943 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7773.21 MB 2025-02-14 11:36:45,943 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35481.71 MB 2025-02-14 11:36:45,943 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35481.71 MB 2025-02-14 11:36:45,943 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:36:45,943 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32622.32 MB 2025-02-14 11:36:45,961 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8140, cut from 8142 2025-02-14 11:36:45,962 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 11:36:45,967 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 11:36:45,967 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 11:36:45,967 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:36:45,968 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:36:45,968 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22344.20 MB 2025-02-14 11:36:45,968 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30760.81 MB 2025-02-14 11:36:45,968 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8416.60 MB 2025-02-14 11:36:45,968 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35481.71 MB 2025-02-14 11:36:45,968 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39665.53 MB 2025-02-14 11:36:45,968 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4183.82 MB 2025-02-14 11:36:45,968 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30760.81 MB 2025-02-14 11:36:46,130 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7932] 2025-02-14 11:36:46,131 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:36:46,131 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 11:36:46,132 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:36:46,132 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 11:36:46,137 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 11:36:46,138 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:36:46,138 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 11:36:46,138 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 11:36:59,523 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:36:59,524 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 11:36:59,529 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 11:36:59,533 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:36:59,533 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1587, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 11:36:59,534 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:36:59,534 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1587, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 11:37:24,310 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 11:37:24,310 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 11:37:24,310 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.77 seconds 2025-02-14 11:37:24,310 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:37:24,310 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24027.18 MB 2025-02-14 11:37:24,310 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29643.48 MB 2025-02-14 11:37:24,310 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5616.30 MB 2025-02-14 11:37:24,310 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48033.17 MB 2025-02-14 11:37:24,310 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39734.74 MB 2025-02-14 11:37:24,310 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8298.43 MB 2025-02-14 11:37:24,310 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38481.38 MB 2025-02-14 11:37:24,400 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 11:37:24,400 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 11:37:24,400 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 11:37:24,400 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:37:24,400 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29643.48 MB 2025-02-14 11:37:24,400 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24028.17 MB 2025-02-14 11:37:24,400 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5615.31 MB 2025-02-14 11:37:24,400 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39734.74 MB 2025-02-14 11:37:24,400 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50780.44 MB 2025-02-14 11:37:24,400 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 11045.70 MB 2025-02-14 11:37:24,400 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45392.52 MB 2025-02-14 11:37:26,333 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 11:37:26,333 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 11:37:26,333 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 11:37:26,333 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:37:26,333 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24028.17 MB 2025-02-14 11:37:26,333 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24559.01 MB 2025-02-14 11:37:26,333 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 11:37:26,333 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50780.44 MB 2025-02-14 11:37:26,333 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29932.65 MB 2025-02-14 11:37:26,333 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20847.79 MB 2025-02-14 11:37:26,333 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28538.34 MB 2025-02-14 11:37:26,349 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 11:37:26,349 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 11:37:26,349 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:37:26,349 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:37:26,349 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24559.01 MB 2025-02-14 11:37:26,349 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26448.54 MB 2025-02-14 11:37:26,349 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 11:37:26,349 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29932.65 MB 2025-02-14 11:37:26,349 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30876.37 MB 2025-02-14 11:37:26,349 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 11:37:26,349 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27865.97 MB 2025-02-14 11:37:26,564 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 11:37:26,565 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 11:37:26,565 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 11:37:26,565 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:37:26,565 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26448.54 MB 2025-02-14 11:37:26,565 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28690.40 MB 2025-02-14 11:37:26,565 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 11:37:26,565 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30876.37 MB 2025-02-14 11:37:26,565 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36538.68 MB 2025-02-14 11:37:26,565 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 11:37:26,565 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34234.68 MB 2025-02-14 11:37:26,565 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 11:37:26,565 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 11:37:26,565 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 11:37:26,565 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:37:26,565 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24559.01 MB 2025-02-14 11:37:26,565 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28690.40 MB 2025-02-14 11:37:26,565 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 11:37:26,565 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29932.65 MB 2025-02-14 11:37:26,565 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36538.68 MB 2025-02-14 11:37:26,566 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 11:37:26,566 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34234.68 MB 2025-02-14 11:37:26,742 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 11:37:26,742 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 11:37:26,742 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 11:37:26,742 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:37:26,742 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30223.94 MB 2025-02-14 11:37:26,742 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30990.94 MB 2025-02-14 11:37:26,742 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 11:37:26,742 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36538.68 MB 2025-02-14 11:37:26,743 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36956.01 MB 2025-02-14 11:37:26,743 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 11:37:26,743 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31698.73 MB 2025-02-14 11:37:26,762 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 11:37:26,762 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 11:37:26,762 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:37:26,762 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:37:26,762 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31403.83 MB 2025-02-14 11:37:26,762 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31632.97 MB 2025-02-14 11:37:26,762 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.13 MB 2025-02-14 11:37:26,762 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36956.01 MB 2025-02-14 11:37:26,762 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36956.01 MB 2025-02-14 11:37:26,762 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:37:26,762 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31835.56 MB 2025-02-14 11:37:26,763 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 11:37:26,763 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 11:37:26,763 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.23 seconds 2025-02-14 11:37:26,763 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:37:26,763 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18497.94 MB 2025-02-14 11:37:26,763 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31834.02 MB 2025-02-14 11:37:26,763 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13336.07 MB 2025-02-14 11:37:26,763 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48033.17 MB 2025-02-14 11:37:26,763 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36956.01 MB 2025-02-14 11:37:26,763 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11077.16 MB 2025-02-14 11:37:26,763 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31835.56 MB 2025-02-14 11:37:27,034 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 11:37:27,034 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 11:37:27,034 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 11:37:27,034 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:37:27,034 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31834.02 MB 2025-02-14 11:37:27,034 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23501.95 MB 2025-02-14 11:37:27,034 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8332.07 MB 2025-02-14 11:37:27,034 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36956.01 MB 2025-02-14 11:37:27,034 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36956.01 MB 2025-02-14 11:37:27,034 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:37:27,034 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34345.38 MB 2025-02-14 11:37:27,052 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8161, cut from 8163 2025-02-14 11:37:27,052 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 11:37:27,058 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 11:37:27,058 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 11:37:27,058 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:37:27,058 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:37:27,058 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23501.95 MB 2025-02-14 11:37:27,058 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31940.79 MB 2025-02-14 11:37:27,059 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8438.84 MB 2025-02-14 11:37:27,059 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36956.01 MB 2025-02-14 11:37:27,059 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45344.62 MB 2025-02-14 11:37:27,059 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8388.61 MB 2025-02-14 11:37:27,059 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31940.79 MB 2025-02-14 11:37:27,226 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7953] 2025-02-14 11:37:27,228 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:37:27,228 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 11:37:27,229 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:37:27,229 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 11:37:27,234 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 11:37:27,235 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:37:27,235 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 11:37:27,235 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 11:37:38,702 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:37:38,702 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 11:37:38,707 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 11:37:38,710 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:37:38,710 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 252, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 11:37:38,711 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:37:38,711 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 252, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 11:37:42,664 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 11:37:42,664 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 11:37:42,664 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.95 seconds 2025-02-14 11:37:42,664 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:37:42,664 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14724.68 MB 2025-02-14 11:37:42,664 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15616.50 MB 2025-02-14 11:37:42,664 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 891.81 MB 2025-02-14 11:37:42,664 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53733.23 MB 2025-02-14 11:37:42,664 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17850.96 MB 2025-02-14 11:37:42,664 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35882.27 MB 2025-02-14 11:37:42,664 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24423.35 MB 2025-02-14 11:37:42,683 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 11:37:42,683 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 11:37:42,683 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:37:42,683 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:37:42,683 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15616.50 MB 2025-02-14 11:37:42,683 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16048.51 MB 2025-02-14 11:37:42,683 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 432.02 MB 2025-02-14 11:37:42,683 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17850.96 MB 2025-02-14 11:37:42,683 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20952.65 MB 2025-02-14 11:37:42,683 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3101.69 MB 2025-02-14 11:37:42,683 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19199.35 MB 2025-02-14 11:37:43,904 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 11:37:43,904 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 11:37:43,904 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.22 seconds 2025-02-14 11:37:43,904 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:37:43,904 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16048.51 MB 2025-02-14 11:37:43,904 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16382.94 MB 2025-02-14 11:37:43,904 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 334.43 MB 2025-02-14 11:37:43,904 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20952.65 MB 2025-02-14 11:37:43,904 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19178.46 MB 2025-02-14 11:37:43,904 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1774.19 MB 2025-02-14 11:37:43,904 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20304.92 MB 2025-02-14 11:37:43,914 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 11:37:43,914 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 11:37:43,914 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:37:43,914 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:37:43,914 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16382.94 MB 2025-02-14 11:37:43,914 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17574.11 MB 2025-02-14 11:37:43,914 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1191.17 MB 2025-02-14 11:37:43,914 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19178.46 MB 2025-02-14 11:37:43,914 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19774.05 MB 2025-02-14 11:37:43,914 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 595.59 MB 2025-02-14 11:37:43,914 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18467.09 MB 2025-02-14 11:37:44,049 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 11:37:44,049 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 11:37:44,049 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 11:37:44,049 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:37:44,049 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17574.11 MB 2025-02-14 11:37:44,049 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18987.55 MB 2025-02-14 11:37:44,049 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1413.44 MB 2025-02-14 11:37:44,049 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19774.05 MB 2025-02-14 11:37:44,049 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23645.39 MB 2025-02-14 11:37:44,049 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3871.34 MB 2025-02-14 11:37:44,050 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22485.14 MB 2025-02-14 11:37:44,050 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 11:37:44,050 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 11:37:44,050 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-14 11:37:44,050 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:37:44,050 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16382.94 MB 2025-02-14 11:37:44,050 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18987.55 MB 2025-02-14 11:37:44,050 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2604.61 MB 2025-02-14 11:37:44,050 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19178.46 MB 2025-02-14 11:37:44,050 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23645.39 MB 2025-02-14 11:37:44,050 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4466.93 MB 2025-02-14 11:37:44,050 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22485.14 MB 2025-02-14 11:37:44,159 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 11:37:44,159 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 11:37:44,159 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 11:37:44,159 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:37:44,159 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19953.68 MB 2025-02-14 11:37:44,159 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20437.42 MB 2025-02-14 11:37:44,159 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 483.74 MB 2025-02-14 11:37:44,159 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23645.39 MB 2025-02-14 11:37:44,159 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23907.53 MB 2025-02-14 11:37:44,159 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 262.14 MB 2025-02-14 11:37:44,159 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20883.32 MB 2025-02-14 11:37:44,172 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 11:37:44,172 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 11:37:44,172 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:37:44,172 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:37:44,173 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20697.54 MB 2025-02-14 11:37:44,173 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20909.90 MB 2025-02-14 11:37:44,173 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 212.36 MB 2025-02-14 11:37:44,173 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23907.53 MB 2025-02-14 11:37:44,173 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23907.53 MB 2025-02-14 11:37:44,173 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:37:44,173 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21015.51 MB 2025-02-14 11:37:44,174 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 11:37:44,174 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 11:37:44,174 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.46 seconds 2025-02-14 11:37:44,174 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:37:44,174 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13846.69 MB 2025-02-14 11:37:44,174 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21110.98 MB 2025-02-14 11:37:44,174 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7264.28 MB 2025-02-14 11:37:44,174 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53733.23 MB 2025-02-14 11:37:44,174 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23907.53 MB 2025-02-14 11:37:44,174 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29825.70 MB 2025-02-14 11:37:44,174 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21110.98 MB 2025-02-14 11:37:44,452 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 11:37:44,452 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 11:37:44,452 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.28 seconds 2025-02-14 11:37:44,452 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:37:44,452 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21110.98 MB 2025-02-14 11:37:44,452 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24125.01 MB 2025-02-14 11:37:44,452 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-14 11:37:44,452 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23907.53 MB 2025-02-14 11:37:44,452 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25786.58 MB 2025-02-14 11:37:44,452 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1879.05 MB 2025-02-14 11:37:44,452 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24426.64 MB 2025-02-14 11:37:44,470 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 11:37:44,470 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 11:37:44,477 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 11:37:44,477 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 11:37:44,477 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:37:44,477 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:37:44,477 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18152.63 MB 2025-02-14 11:37:44,477 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26591.65 MB 2025-02-14 11:37:44,477 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 11:37:44,477 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25786.58 MB 2025-02-14 11:37:44,477 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36276.54 MB 2025-02-14 11:37:44,477 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 11:37:44,477 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26591.65 MB 2025-02-14 11:37:44,640 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 11:37:44,641 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:37:44,641 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 11:37:44,642 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:37:44,642 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 11:37:44,647 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 11:37:44,648 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:37:44,648 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 11:37:44,648 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 11:37:53,496 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:37:53,496 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 11:37:53,501 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 11:37:53,505 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:37:53,505 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 231, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 11:37:53,506 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:37:53,506 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 231, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 11:37:57,156 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 11:37:57,156 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 11:37:57,156 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.65 seconds 2025-02-14 11:37:57,156 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:37:57,156 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14578.35 MB 2025-02-14 11:37:57,156 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15395.85 MB 2025-02-14 11:37:57,156 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 817.50 MB 2025-02-14 11:37:57,156 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48861.54 MB 2025-02-14 11:37:57,156 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17939.04 MB 2025-02-14 11:37:57,156 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30922.51 MB 2025-02-14 11:37:57,156 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24277.02 MB 2025-02-14 11:37:57,172 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 11:37:57,172 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 11:37:57,172 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:37:57,172 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:37:57,172 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15395.85 MB 2025-02-14 11:37:57,172 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15750.37 MB 2025-02-14 11:37:57,172 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 354.53 MB 2025-02-14 11:37:57,172 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17939.04 MB 2025-02-14 11:37:57,172 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19918.75 MB 2025-02-14 11:37:57,172 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1979.71 MB 2025-02-14 11:37:57,172 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18582.97 MB 2025-02-14 11:37:58,264 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 11:37:58,264 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 11:37:58,264 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.09 seconds 2025-02-14 11:37:58,264 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:37:58,264 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15750.37 MB 2025-02-14 11:37:58,264 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16048.97 MB 2025-02-14 11:37:58,264 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 298.60 MB 2025-02-14 11:37:58,264 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19918.75 MB 2025-02-14 11:37:58,264 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19138.61 MB 2025-02-14 11:37:58,264 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -780.14 MB 2025-02-14 11:37:58,264 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20007.82 MB 2025-02-14 11:37:58,273 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 11:37:58,273 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 11:37:58,273 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:37:58,273 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:37:58,273 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16048.97 MB 2025-02-14 11:37:58,273 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17111.58 MB 2025-02-14 11:37:58,273 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1062.60 MB 2025-02-14 11:37:58,273 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19138.61 MB 2025-02-14 11:37:58,273 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19671.29 MB 2025-02-14 11:37:58,273 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 532.68 MB 2025-02-14 11:37:58,273 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17908.88 MB 2025-02-14 11:37:58,394 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 11:37:58,394 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 11:37:58,394 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 11:37:58,394 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:37:58,394 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17111.58 MB 2025-02-14 11:37:58,394 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18373.10 MB 2025-02-14 11:37:58,394 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1261.52 MB 2025-02-14 11:37:58,394 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19671.29 MB 2025-02-14 11:37:58,394 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23133.68 MB 2025-02-14 11:37:58,394 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3462.40 MB 2025-02-14 11:37:58,394 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21492.65 MB 2025-02-14 11:37:58,395 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 11:37:58,395 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 11:37:58,395 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 11:37:58,395 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:37:58,395 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16048.97 MB 2025-02-14 11:37:58,395 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18373.10 MB 2025-02-14 11:37:58,395 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2324.13 MB 2025-02-14 11:37:58,395 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19138.61 MB 2025-02-14 11:37:58,395 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23133.68 MB 2025-02-14 11:37:58,395 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3995.07 MB 2025-02-14 11:37:58,395 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21492.65 MB 2025-02-14 11:37:58,490 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 11:37:58,490 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 11:37:58,490 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 11:37:58,490 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:37:58,490 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19235.72 MB 2025-02-14 11:37:58,490 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19668.07 MB 2025-02-14 11:37:58,490 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 432.36 MB 2025-02-14 11:37:58,490 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23133.68 MB 2025-02-14 11:37:58,490 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23364.37 MB 2025-02-14 11:37:58,490 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 230.69 MB 2025-02-14 11:37:58,490 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20066.31 MB 2025-02-14 11:37:58,502 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 11:37:58,502 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 11:37:58,502 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:37:58,502 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:37:58,502 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19900.33 MB 2025-02-14 11:37:58,502 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20128.66 MB 2025-02-14 11:37:58,502 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.33 MB 2025-02-14 11:37:58,502 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23364.37 MB 2025-02-14 11:37:58,502 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23366.47 MB 2025-02-14 11:37:58,502 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-14 11:37:58,502 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20206.16 MB 2025-02-14 11:37:58,503 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 11:37:58,503 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 11:37:58,503 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.00 seconds 2025-02-14 11:37:58,503 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:37:58,503 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13773.53 MB 2025-02-14 11:37:58,503 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20329.74 MB 2025-02-14 11:37:58,503 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6556.21 MB 2025-02-14 11:37:58,503 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48861.54 MB 2025-02-14 11:37:58,503 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23366.47 MB 2025-02-14 11:37:58,503 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25495.08 MB 2025-02-14 11:37:58,503 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20329.74 MB 2025-02-14 11:37:58,773 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 11:37:58,773 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 11:37:58,773 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 11:37:58,773 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:37:58,773 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14938.66 MB 2025-02-14 11:37:58,773 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17952.70 MB 2025-02-14 11:37:58,773 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-14 11:37:58,773 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23366.47 MB 2025-02-14 11:37:58,773 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23366.47 MB 2025-02-14 11:37:58,773 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:37:58,773 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18254.06 MB 2025-02-14 11:37:58,792 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 11:37:58,792 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 11:37:58,798 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 11:37:58,798 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 11:37:58,798 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:37:58,798 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:37:58,798 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17952.70 MB 2025-02-14 11:37:58,798 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26391.72 MB 2025-02-14 11:37:58,798 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 11:37:58,798 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23366.47 MB 2025-02-14 11:37:58,798 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33856.42 MB 2025-02-14 11:37:58,798 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 11:37:58,798 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26391.72 MB 2025-02-14 11:37:58,965 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 11:37:58,966 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:37:58,966 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 11:37:58,967 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:37:58,967 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 11:37:58,972 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 11:37:58,973 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:37:58,973 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 11:37:58,973 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 11:39:29,179 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:39:29,180 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 11:39:29,184 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 11:39:29,189 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:39:29,189 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 149, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 11:39:29,190 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:39:29,190 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 149, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 11:39:31,558 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 11:39:31,558 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 11:39:31,558 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.36 seconds 2025-02-14 11:39:31,558 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:39:31,558 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14006.96 MB 2025-02-14 11:39:31,558 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14534.26 MB 2025-02-14 11:39:31,558 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 527.30 MB 2025-02-14 11:39:31,558 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46441.43 MB 2025-02-14 11:39:31,558 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18010.34 MB 2025-02-14 11:39:31,558 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28431.09 MB 2025-02-14 11:39:31,558 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23479.14 MB 2025-02-14 11:39:31,573 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 11:39:31,573 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 11:39:31,573 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:39:31,573 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:39:31,573 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14534.26 MB 2025-02-14 11:39:31,573 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14761.65 MB 2025-02-14 11:39:31,573 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.38 MB 2025-02-14 11:39:31,573 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18010.34 MB 2025-02-14 11:39:31,573 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18010.34 MB 2025-02-14 11:39:31,573 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:39:31,573 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16588.70 MB 2025-02-14 11:39:32,296 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 11:39:32,296 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 11:39:32,296 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.72 seconds 2025-02-14 11:39:32,296 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:39:32,296 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14761.65 MB 2025-02-14 11:39:32,296 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14954.08 MB 2025-02-14 11:39:32,296 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 192.43 MB 2025-02-14 11:39:32,296 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18010.34 MB 2025-02-14 11:39:32,296 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17607.69 MB 2025-02-14 11:39:32,296 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -402.65 MB 2025-02-14 11:39:32,296 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18933.12 MB 2025-02-14 11:39:32,306 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 11:39:32,306 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 11:39:32,306 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 11:39:32,306 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:39:32,306 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14954.01 MB 2025-02-14 11:39:32,306 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15638.80 MB 2025-02-14 11:39:32,306 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 684.79 MB 2025-02-14 11:39:32,306 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17607.69 MB 2025-02-14 11:39:32,306 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17607.69 MB 2025-02-14 11:39:32,306 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:39:32,306 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16152.63 MB 2025-02-14 11:39:32,408 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 11:39:32,408 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 11:39:32,408 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 11:39:32,408 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:39:32,408 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15638.80 MB 2025-02-14 11:39:32,408 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16451.52 MB 2025-02-14 11:39:32,408 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 812.71 MB 2025-02-14 11:39:32,408 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17607.69 MB 2025-02-14 11:39:32,408 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19327.35 MB 2025-02-14 11:39:32,408 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1719.66 MB 2025-02-14 11:39:32,408 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18461.28 MB 2025-02-14 11:39:32,409 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 11:39:32,409 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 11:39:32,409 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 11:39:32,409 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:39:32,409 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14954.01 MB 2025-02-14 11:39:32,409 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16451.52 MB 2025-02-14 11:39:32,410 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1497.50 MB 2025-02-14 11:39:32,410 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17607.69 MB 2025-02-14 11:39:32,410 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19327.35 MB 2025-02-14 11:39:32,410 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1719.66 MB 2025-02-14 11:39:32,410 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18461.28 MB 2025-02-14 11:39:32,505 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 11:39:32,505 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 11:39:32,505 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 11:39:32,505 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:39:32,506 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17007.43 MB 2025-02-14 11:39:32,506 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17285.47 MB 2025-02-14 11:39:32,506 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 278.04 MB 2025-02-14 11:39:32,506 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19327.35 MB 2025-02-14 11:39:32,506 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19476.25 MB 2025-02-14 11:39:32,506 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 148.90 MB 2025-02-14 11:39:32,506 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17553.83 MB 2025-02-14 11:39:32,519 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 11:39:32,519 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 11:39:32,519 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:39:32,519 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:39:32,519 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17435.15 MB 2025-02-14 11:39:32,519 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17663.37 MB 2025-02-14 11:39:32,519 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.23 MB 2025-02-14 11:39:32,519 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19476.25 MB 2025-02-14 11:39:32,519 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19476.25 MB 2025-02-14 11:39:32,519 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:39:32,519 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17663.37 MB 2025-02-14 11:39:32,521 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 11:39:32,521 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 11:39:32,521 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.33 seconds 2025-02-14 11:39:32,521 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:39:32,521 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13487.83 MB 2025-02-14 11:39:32,521 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17864.03 MB 2025-02-14 11:39:32,521 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4376.19 MB 2025-02-14 11:39:32,521 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46441.43 MB 2025-02-14 11:39:32,521 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19476.25 MB 2025-02-14 11:39:32,521 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26965.18 MB 2025-02-14 11:39:32,521 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17864.03 MB 2025-02-14 11:39:32,791 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 11:39:32,791 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 11:39:32,791 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 11:39:32,791 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:39:32,791 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17864.03 MB 2025-02-14 11:39:32,791 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17282.33 MB 2025-02-14 11:39:32,791 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -581.70 MB 2025-02-14 11:39:32,791 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19476.25 MB 2025-02-14 11:39:32,791 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19744.69 MB 2025-02-14 11:39:32,791 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 268.44 MB 2025-02-14 11:39:32,791 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18967.41 MB 2025-02-14 11:39:32,809 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8145, cut from 8147 2025-02-14 11:39:32,810 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2,'] 2025-02-14 11:39:32,816 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 11:39:32,816 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 11:39:32,816 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:39:32,816 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:39:32,816 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17282.33 MB 2025-02-14 11:39:32,816 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25704.51 MB 2025-02-14 11:39:32,816 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8422.18 MB 2025-02-14 11:39:32,816 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19744.69 MB 2025-02-14 11:39:32,816 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30209.47 MB 2025-02-14 11:39:32,816 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10464.79 MB 2025-02-14 11:39:32,816 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25704.51 MB 2025-02-14 11:39:32,987 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7937] 2025-02-14 11:39:32,989 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:39:32,989 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 11:39:32,990 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:39:32,990 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 11:39:32,995 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 11:39:32,996 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:39:32,996 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 11:39:32,996 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2,'] 2025-02-14 11:41:35,649 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:41:35,650 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 11:41:35,658 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 11:41:35,665 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:41:35,666 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2064, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 11:41:35,667 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:41:35,668 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2064, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 11:42:07,393 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 11:42:07,394 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 11:42:07,394 - resource_logging.py:150 - __exit__ - DEBUG - Time: 31.71 seconds 2025-02-14 11:42:07,394 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:42:07,394 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27350.99 MB 2025-02-14 11:42:07,394 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34655.37 MB 2025-02-14 11:42:07,394 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7304.38 MB 2025-02-14 11:42:07,394 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38581.31 MB 2025-02-14 11:42:07,394 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40957.38 MB 2025-02-14 11:42:07,394 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2376.07 MB 2025-02-14 11:42:07,394 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43617.94 MB 2025-02-14 11:42:07,521 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 11:42:07,521 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 11:42:07,521 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 11:42:07,521 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:42:07,521 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34655.37 MB 2025-02-14 11:42:07,521 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26507.94 MB 2025-02-14 11:42:07,521 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8147.43 MB 2025-02-14 11:42:07,521 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40957.38 MB 2025-02-14 11:42:07,521 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 64367.89 MB 2025-02-14 11:42:07,521 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 23410.51 MB 2025-02-14 11:42:07,521 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54607.32 MB 2025-02-14 11:42:09,458 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 11:42:09,458 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 11:42:09,458 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 11:42:09,458 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:42:09,458 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26507.94 MB 2025-02-14 11:42:09,458 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27038.78 MB 2025-02-14 11:42:09,458 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 11:42:09,458 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64367.89 MB 2025-02-14 11:42:09,458 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30882.66 MB 2025-02-14 11:42:09,458 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33485.23 MB 2025-02-14 11:42:09,458 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31019.16 MB 2025-02-14 11:42:09,472 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 11:42:09,472 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 11:42:09,472 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:42:09,472 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:42:09,472 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27038.78 MB 2025-02-14 11:42:09,472 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28928.32 MB 2025-02-14 11:42:09,472 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 11:42:09,472 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30882.66 MB 2025-02-14 11:42:09,472 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33713.82 MB 2025-02-14 11:42:09,472 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-14 11:42:09,472 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30345.75 MB 2025-02-14 11:42:09,682 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 11:42:09,682 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 11:42:09,682 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 11:42:09,682 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:42:09,683 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28928.32 MB 2025-02-14 11:42:09,683 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31170.17 MB 2025-02-14 11:42:09,683 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 11:42:09,683 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33713.82 MB 2025-02-14 11:42:09,683 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39376.13 MB 2025-02-14 11:42:09,683 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 11:42:09,683 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36714.46 MB 2025-02-14 11:42:09,683 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 11:42:09,683 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 11:42:09,683 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 11:42:09,683 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:42:09,683 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27038.78 MB 2025-02-14 11:42:09,683 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31170.17 MB 2025-02-14 11:42:09,683 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 11:42:09,683 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30882.66 MB 2025-02-14 11:42:09,683 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39376.13 MB 2025-02-14 11:42:09,683 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8493.47 MB 2025-02-14 11:42:09,683 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36714.46 MB 2025-02-14 11:42:09,858 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 11:42:09,858 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 11:42:09,858 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 11:42:09,858 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:42:09,858 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32703.72 MB 2025-02-14 11:42:09,858 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33470.72 MB 2025-02-14 11:42:09,858 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 11:42:09,858 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39376.13 MB 2025-02-14 11:42:09,858 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39791.36 MB 2025-02-14 11:42:09,858 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 11:42:09,858 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34178.51 MB 2025-02-14 11:42:09,879 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 11:42:09,879 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 11:42:09,879 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:42:09,879 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:42:09,879 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33883.61 MB 2025-02-14 11:42:09,879 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34111.71 MB 2025-02-14 11:42:09,879 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.10 MB 2025-02-14 11:42:09,879 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39791.36 MB 2025-02-14 11:42:09,879 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39791.36 MB 2025-02-14 11:42:09,879 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:42:09,879 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34318.50 MB 2025-02-14 11:42:09,880 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 11:42:09,880 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 11:42:09,880 - resource_logging.py:150 - __exit__ - DEBUG - Time: 34.21 seconds 2025-02-14 11:42:09,880 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:42:09,880 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20159.85 MB 2025-02-14 11:42:09,880 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34311.72 MB 2025-02-14 11:42:09,880 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14151.88 MB 2025-02-14 11:42:09,880 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38581.31 MB 2025-02-14 11:42:09,880 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39791.36 MB 2025-02-14 11:42:09,880 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1210.06 MB 2025-02-14 11:42:09,880 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34318.50 MB 2025-02-14 11:42:10,149 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 11:42:10,149 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 11:42:10,149 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 11:42:10,149 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:42:10,149 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34311.72 MB 2025-02-14 11:42:10,149 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25147.86 MB 2025-02-14 11:42:10,149 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9163.87 MB 2025-02-14 11:42:10,149 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39791.36 MB 2025-02-14 11:42:10,149 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39791.36 MB 2025-02-14 11:42:10,149 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:42:10,149 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36810.18 MB 2025-02-14 11:42:10,167 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8119, cut from 8121 2025-02-14 11:42:10,167 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 11:42:10,173 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 11:42:10,173 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 11:42:10,173 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:42:10,173 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:42:10,173 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25147.86 MB 2025-02-14 11:42:10,173 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33543.07 MB 2025-02-14 11:42:10,173 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8395.21 MB 2025-02-14 11:42:10,173 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39791.36 MB 2025-02-14 11:42:10,173 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48138.03 MB 2025-02-14 11:42:10,173 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8346.66 MB 2025-02-14 11:42:10,173 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33543.07 MB 2025-02-14 11:42:10,336 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7911] 2025-02-14 11:42:10,337 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:42:10,337 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 11:42:10,338 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:42:10,338 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 11:42:10,343 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 11:42:10,344 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:42:10,344 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 11:42:10,344 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 11:42:39,566 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:42:39,566 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 11:42:39,571 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 11:42:39,576 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:42:39,576 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2771, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 11:42:39,577 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:42:39,577 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2771, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 11:43:22,675 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 11:43:22,675 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 11:43:22,675 - resource_logging.py:150 - __exit__ - DEBUG - Time: 43.09 seconds 2025-02-14 11:43:22,675 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:43:22,675 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32279.41 MB 2025-02-14 11:43:22,675 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42085.83 MB 2025-02-14 11:43:22,675 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 9806.41 MB 2025-02-14 11:43:22,675 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 75799.46 MB 2025-02-14 11:43:22,675 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46032.49 MB 2025-02-14 11:43:22,675 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29766.98 MB 2025-02-14 11:43:22,675 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51892.24 MB 2025-02-14 11:43:22,859 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 11:43:22,859 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 11:43:22,859 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-14 11:43:22,859 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:43:22,859 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42085.83 MB 2025-02-14 11:43:22,859 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30185.35 MB 2025-02-14 11:43:22,859 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -11900.48 MB 2025-02-14 11:43:22,860 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46032.49 MB 2025-02-14 11:43:22,860 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 82537.61 MB 2025-02-14 11:43:22,860 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 36505.12 MB 2025-02-14 11:43:22,860 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 69941.95 MB 2025-02-14 11:43:24,850 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 11:43:24,850 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 11:43:24,850 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.99 seconds 2025-02-14 11:43:24,850 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:43:24,850 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30185.35 MB 2025-02-14 11:43:24,850 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30716.19 MB 2025-02-14 11:43:24,850 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 11:43:24,850 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 82537.61 MB 2025-02-14 11:43:24,850 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32734.45 MB 2025-02-14 11:43:24,850 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -49803.17 MB 2025-02-14 11:43:24,850 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34696.56 MB 2025-02-14 11:43:24,864 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 11:43:24,864 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 11:43:24,864 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:43:24,864 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:43:24,864 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30716.19 MB 2025-02-14 11:43:24,864 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32605.66 MB 2025-02-14 11:43:24,864 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.47 MB 2025-02-14 11:43:24,864 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32734.45 MB 2025-02-14 11:43:24,864 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36037.46 MB 2025-02-14 11:43:24,864 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 11:43:24,864 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34023.09 MB 2025-02-14 11:43:25,073 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 11:43:25,073 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 11:43:25,073 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 11:43:25,073 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:43:25,073 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32605.66 MB 2025-02-14 11:43:25,073 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34847.52 MB 2025-02-14 11:43:25,073 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 11:43:25,073 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36037.46 MB 2025-02-14 11:43:25,073 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42643.49 MB 2025-02-14 11:43:25,073 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 11:43:25,073 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40391.80 MB 2025-02-14 11:43:25,074 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 11:43:25,074 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 11:43:25,074 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 11:43:25,074 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:43:25,074 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30716.19 MB 2025-02-14 11:43:25,074 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34847.52 MB 2025-02-14 11:43:25,074 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.32 MB 2025-02-14 11:43:25,074 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32734.45 MB 2025-02-14 11:43:25,074 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42643.49 MB 2025-02-14 11:43:25,074 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9909.04 MB 2025-02-14 11:43:25,074 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40391.80 MB 2025-02-14 11:43:25,278 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 11:43:25,278 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 11:43:25,278 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 11:43:25,278 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:43:25,278 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36381.06 MB 2025-02-14 11:43:25,278 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37148.06 MB 2025-02-14 11:43:25,278 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 11:43:25,278 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42643.49 MB 2025-02-14 11:43:25,278 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43058.72 MB 2025-02-14 11:43:25,278 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 11:43:25,278 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37855.85 MB 2025-02-14 11:43:25,297 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 11:43:25,297 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 11:43:25,297 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:43:25,297 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:43:25,297 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37560.95 MB 2025-02-14 11:43:25,297 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37789.81 MB 2025-02-14 11:43:25,297 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.86 MB 2025-02-14 11:43:25,297 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43058.72 MB 2025-02-14 11:43:25,297 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43058.72 MB 2025-02-14 11:43:25,297 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:43:25,297 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38011.40 MB 2025-02-14 11:43:25,298 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 11:43:25,298 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 11:43:25,298 - resource_logging.py:150 - __exit__ - DEBUG - Time: 45.72 seconds 2025-02-14 11:43:25,298 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:43:25,298 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22624.06 MB 2025-02-14 11:43:25,298 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37990.79 MB 2025-02-14 11:43:25,298 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 15366.73 MB 2025-02-14 11:43:25,298 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 66142.08 MB 2025-02-14 11:43:25,298 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43058.72 MB 2025-02-14 11:43:25,298 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23083.35 MB 2025-02-14 11:43:25,298 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38011.40 MB 2025-02-14 11:43:25,568 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 11:43:25,568 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 11:43:25,568 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 11:43:25,568 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:43:25,568 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37990.79 MB 2025-02-14 11:43:25,568 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27626.93 MB 2025-02-14 11:43:25,568 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10363.86 MB 2025-02-14 11:43:25,568 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43058.72 MB 2025-02-14 11:43:25,568 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43058.72 MB 2025-02-14 11:43:25,568 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:43:25,568 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40501.22 MB 2025-02-14 11:43:25,586 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8158, cut from 8160 2025-02-14 11:43:25,586 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 11:43:25,592 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 11:43:25,592 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 11:43:25,592 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:43:25,592 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:43:25,592 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27626.93 MB 2025-02-14 11:43:25,592 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36061.49 MB 2025-02-14 11:43:25,592 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8434.56 MB 2025-02-14 11:43:25,592 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43058.72 MB 2025-02-14 11:43:25,592 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47253.03 MB 2025-02-14 11:43:25,592 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4194.30 MB 2025-02-14 11:43:25,592 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36061.49 MB 2025-02-14 11:43:25,754 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7950] 2025-02-14 11:43:25,756 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:43:25,756 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 11:43:25,757 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:43:25,757 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 11:43:25,761 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 11:43:25,762 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:43:25,762 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 11:43:25,763 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 11:44:24,668 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:44:24,668 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 11:44:24,676 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 11:44:24,684 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:44:24,684 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 640, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 11:44:24,686 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:44:24,686 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 640, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 11:44:34,635 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 11:44:34,635 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 11:44:34,635 - resource_logging.py:150 - __exit__ - DEBUG - Time: 9.94 seconds 2025-02-14 11:44:34,635 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:44:34,635 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17428.33 MB 2025-02-14 11:44:34,635 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19693.25 MB 2025-02-14 11:44:34,635 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2264.92 MB 2025-02-14 11:44:34,635 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55637.44 MB 2025-02-14 11:44:34,635 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24782.05 MB 2025-02-14 11:44:34,635 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30855.40 MB 2025-02-14 11:44:34,635 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28711.64 MB 2025-02-14 11:44:34,685 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 11:44:34,685 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 11:44:34,685 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-14 11:44:34,685 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:44:34,685 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19693.25 MB 2025-02-14 11:44:34,685 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19106.06 MB 2025-02-14 11:44:34,685 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -587.19 MB 2025-02-14 11:44:34,685 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24782.05 MB 2025-02-14 11:44:34,685 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31067.21 MB 2025-02-14 11:44:34,685 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6285.16 MB 2025-02-14 11:44:34,685 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28135.18 MB 2025-02-14 11:44:36,609 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 11:44:36,609 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 11:44:36,609 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 11:44:36,609 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:44:36,609 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19106.06 MB 2025-02-14 11:44:36,609 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19636.90 MB 2025-02-14 11:44:36,609 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 11:44:36,609 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31067.21 MB 2025-02-14 11:44:36,609 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24641.54 MB 2025-02-14 11:44:36,609 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6425.67 MB 2025-02-14 11:44:36,609 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23616.23 MB 2025-02-14 11:44:36,623 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 11:44:36,623 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 11:44:36,623 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:44:36,623 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:44:36,623 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19636.90 MB 2025-02-14 11:44:36,623 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21526.43 MB 2025-02-14 11:44:36,623 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 11:44:36,623 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24641.54 MB 2025-02-14 11:44:36,623 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25585.25 MB 2025-02-14 11:44:36,623 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 11:44:36,623 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22943.86 MB 2025-02-14 11:44:36,832 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 11:44:36,832 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 11:44:36,832 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 11:44:36,832 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:44:36,832 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21526.43 MB 2025-02-14 11:44:36,832 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23768.29 MB 2025-02-14 11:44:36,832 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 11:44:36,832 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25585.25 MB 2025-02-14 11:44:36,832 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31719.42 MB 2025-02-14 11:44:36,833 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 11:44:36,833 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29312.57 MB 2025-02-14 11:44:36,833 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 11:44:36,833 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 11:44:36,833 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 11:44:36,833 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:44:36,833 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19636.90 MB 2025-02-14 11:44:36,833 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23768.29 MB 2025-02-14 11:44:36,833 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 11:44:36,833 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24641.54 MB 2025-02-14 11:44:36,833 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31719.42 MB 2025-02-14 11:44:36,833 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7077.89 MB 2025-02-14 11:44:36,833 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29312.57 MB 2025-02-14 11:44:37,001 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 11:44:37,001 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 11:44:37,001 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 11:44:37,001 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:44:37,001 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25301.83 MB 2025-02-14 11:44:37,001 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26068.83 MB 2025-02-14 11:44:37,001 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 11:44:37,002 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31719.42 MB 2025-02-14 11:44:37,002 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32134.66 MB 2025-02-14 11:44:37,002 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 11:44:37,002 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26776.62 MB 2025-02-14 11:44:37,020 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 11:44:37,020 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 11:44:37,020 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:44:37,020 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:44:37,020 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26481.72 MB 2025-02-14 11:44:37,020 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26711.13 MB 2025-02-14 11:44:37,020 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.40 MB 2025-02-14 11:44:37,020 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32134.66 MB 2025-02-14 11:44:37,020 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32134.66 MB 2025-02-14 11:44:37,020 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:44:37,020 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26901.04 MB 2025-02-14 11:44:37,021 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 11:44:37,022 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 11:44:37,022 - resource_logging.py:150 - __exit__ - DEBUG - Time: 12.33 seconds 2025-02-14 11:44:37,022 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:44:37,022 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15198.52 MB 2025-02-14 11:44:37,022 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26912.20 MB 2025-02-14 11:44:37,022 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11713.68 MB 2025-02-14 11:44:37,022 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55637.44 MB 2025-02-14 11:44:37,022 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32134.66 MB 2025-02-14 11:44:37,022 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23502.78 MB 2025-02-14 11:44:37,022 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26912.20 MB 2025-02-14 11:44:37,289 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 11:44:37,289 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 11:44:37,289 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 11:44:37,289 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:44:37,289 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26912.20 MB 2025-02-14 11:44:37,289 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20202.91 MB 2025-02-14 11:44:37,289 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6709.29 MB 2025-02-14 11:44:37,289 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32134.66 MB 2025-02-14 11:44:37,289 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32134.66 MB 2025-02-14 11:44:37,290 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:44:37,290 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29423.87 MB 2025-02-14 11:44:37,307 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 11:44:37,308 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 11:44:37,314 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 11:44:37,314 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 11:44:37,314 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:44:37,314 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:44:37,314 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20202.91 MB 2025-02-14 11:44:37,314 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28641.93 MB 2025-02-14 11:44:37,314 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 11:44:37,314 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32134.66 MB 2025-02-14 11:44:37,314 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40525.37 MB 2025-02-14 11:44:37,314 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 11:44:37,314 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28641.93 MB 2025-02-14 11:44:37,482 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 11:44:37,483 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:44:37,483 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 11:44:37,484 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:44:37,484 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 11:44:37,489 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 11:44:37,490 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:44:37,490 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 11:44:37,491 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 11:45:36,496 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:45:36,496 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 11:45:36,501 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 11:45:36,505 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:45:36,505 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1579, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 11:45:36,506 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:45:36,506 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1579, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 11:46:00,762 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 11:46:00,762 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 11:46:00,762 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.25 seconds 2025-02-14 11:46:00,762 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:46:00,762 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23971.43 MB 2025-02-14 11:46:00,762 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29560.34 MB 2025-02-14 11:46:00,762 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5588.91 MB 2025-02-14 11:46:00,763 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53110.37 MB 2025-02-14 11:46:00,763 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39273.37 MB 2025-02-14 11:46:00,763 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13837.01 MB 2025-02-14 11:46:00,763 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38426.44 MB 2025-02-14 11:46:00,831 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 11:46:00,831 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 11:46:00,831 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 11:46:00,831 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:46:00,831 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29560.34 MB 2025-02-14 11:46:00,831 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23986.58 MB 2025-02-14 11:46:00,831 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5573.76 MB 2025-02-14 11:46:00,831 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39273.37 MB 2025-02-14 11:46:00,831 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43234.89 MB 2025-02-14 11:46:00,831 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3961.52 MB 2025-02-14 11:46:00,831 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38873.61 MB 2025-02-14 11:46:02,735 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 11:46:02,736 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 11:46:02,736 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.90 seconds 2025-02-14 11:46:02,736 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:46:02,736 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23986.58 MB 2025-02-14 11:46:02,736 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24517.42 MB 2025-02-14 11:46:02,736 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 11:46:02,736 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43234.89 MB 2025-02-14 11:46:02,736 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30905.73 MB 2025-02-14 11:46:02,736 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12329.16 MB 2025-02-14 11:46:02,736 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28496.75 MB 2025-02-14 11:46:02,749 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 11:46:02,749 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 11:46:02,749 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:46:02,749 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:46:02,749 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24517.42 MB 2025-02-14 11:46:02,749 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26406.96 MB 2025-02-14 11:46:02,749 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 11:46:02,749 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30905.73 MB 2025-02-14 11:46:02,749 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30905.73 MB 2025-02-14 11:46:02,749 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:46:02,749 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27824.38 MB 2025-02-14 11:46:02,959 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 11:46:02,959 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 11:46:02,959 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 11:46:02,959 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:46:02,959 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26406.96 MB 2025-02-14 11:46:02,959 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28648.81 MB 2025-02-14 11:46:02,959 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 11:46:02,959 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30905.73 MB 2025-02-14 11:46:02,959 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36568.04 MB 2025-02-14 11:46:02,959 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 11:46:02,959 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34193.09 MB 2025-02-14 11:46:02,960 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 11:46:02,960 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 11:46:02,960 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 11:46:02,960 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:46:02,960 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24517.42 MB 2025-02-14 11:46:02,960 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28648.81 MB 2025-02-14 11:46:02,960 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 11:46:02,960 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30905.73 MB 2025-02-14 11:46:02,960 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36568.04 MB 2025-02-14 11:46:02,960 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 11:46:02,960 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34193.09 MB 2025-02-14 11:46:03,131 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 11:46:03,131 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 11:46:03,131 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 11:46:03,131 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:46:03,131 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30182.35 MB 2025-02-14 11:46:03,131 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30949.36 MB 2025-02-14 11:46:03,131 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 11:46:03,131 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36568.04 MB 2025-02-14 11:46:03,131 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36983.28 MB 2025-02-14 11:46:03,131 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 11:46:03,131 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31657.14 MB 2025-02-14 11:46:03,150 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 11:46:03,150 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 11:46:03,150 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:46:03,150 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:46:03,150 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31362.24 MB 2025-02-14 11:46:03,150 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31591.37 MB 2025-02-14 11:46:03,150 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.12 MB 2025-02-14 11:46:03,150 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36983.28 MB 2025-02-14 11:46:03,150 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36983.28 MB 2025-02-14 11:46:03,150 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:46:03,150 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31782.52 MB 2025-02-14 11:46:03,151 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 11:46:03,151 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 11:46:03,151 - resource_logging.py:150 - __exit__ - DEBUG - Time: 26.64 seconds 2025-02-14 11:46:03,151 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:46:03,151 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18470.07 MB 2025-02-14 11:46:03,151 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31792.44 MB 2025-02-14 11:46:03,151 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13322.37 MB 2025-02-14 11:46:03,151 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53110.37 MB 2025-02-14 11:46:03,151 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36983.28 MB 2025-02-14 11:46:03,151 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16127.10 MB 2025-02-14 11:46:03,151 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31792.44 MB 2025-02-14 11:46:03,419 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 11:46:03,419 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 11:46:03,419 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 11:46:03,419 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:46:03,419 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31792.44 MB 2025-02-14 11:46:03,419 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23474.59 MB 2025-02-14 11:46:03,420 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8317.85 MB 2025-02-14 11:46:03,420 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36983.28 MB 2025-02-14 11:46:03,420 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36983.28 MB 2025-02-14 11:46:03,420 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:46:03,420 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34304.10 MB 2025-02-14 11:46:03,437 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 11:46:03,438 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 11:46:03,444 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 11:46:03,444 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 11:46:03,444 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:46:03,444 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:46:03,444 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23474.59 MB 2025-02-14 11:46:03,444 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31913.61 MB 2025-02-14 11:46:03,444 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 11:46:03,444 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36983.28 MB 2025-02-14 11:46:03,444 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45373.98 MB 2025-02-14 11:46:03,444 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 11:46:03,444 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31913.61 MB 2025-02-14 11:46:03,606 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 11:46:03,608 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:46:03,608 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 11:46:03,609 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:46:03,609 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 11:46:03,613 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 11:46:03,614 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:46:03,614 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 11:46:03,615 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 11:46:27,682 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:46:27,682 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 11:46:27,687 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 11:46:27,691 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:46:27,691 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1322, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 11:46:27,692 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:46:27,692 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1322, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 11:46:48,151 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 11:46:48,151 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 11:46:48,151 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.45 seconds 2025-02-14 11:46:48,151 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:46:48,151 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22180.61 MB 2025-02-14 11:46:48,151 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26859.36 MB 2025-02-14 11:46:48,151 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4678.75 MB 2025-02-14 11:46:48,151 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57958.99 MB 2025-02-14 11:46:48,151 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38363.20 MB 2025-02-14 11:46:48,151 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19595.79 MB 2025-02-14 11:46:48,151 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35728.85 MB 2025-02-14 11:46:48,228 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 11:46:48,229 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 11:46:48,229 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 11:46:48,229 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:46:48,229 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26859.36 MB 2025-02-14 11:46:48,229 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22650.52 MB 2025-02-14 11:46:48,229 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4208.84 MB 2025-02-14 11:46:48,229 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38363.20 MB 2025-02-14 11:46:48,229 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47559.21 MB 2025-02-14 11:46:48,229 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9196.01 MB 2025-02-14 11:46:48,229 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40720.54 MB 2025-02-14 11:46:50,156 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 11:46:50,156 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 11:46:50,156 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 11:46:50,156 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:46:50,156 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22650.52 MB 2025-02-14 11:46:50,156 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23181.36 MB 2025-02-14 11:46:50,156 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 11:46:50,156 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47559.21 MB 2025-02-14 11:46:50,156 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29492.25 MB 2025-02-14 11:46:50,156 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18066.96 MB 2025-02-14 11:46:50,156 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27160.69 MB 2025-02-14 11:46:50,169 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 11:46:50,169 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 11:46:50,169 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:46:50,169 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:46:50,169 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23181.36 MB 2025-02-14 11:46:50,169 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25070.89 MB 2025-02-14 11:46:50,169 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 11:46:50,169 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29492.25 MB 2025-02-14 11:46:50,169 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29492.25 MB 2025-02-14 11:46:50,169 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:46:50,169 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26488.32 MB 2025-02-14 11:46:50,383 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 11:46:50,383 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 11:46:50,383 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 11:46:50,383 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:46:50,383 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25070.89 MB 2025-02-14 11:46:50,383 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27312.75 MB 2025-02-14 11:46:50,383 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 11:46:50,383 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29492.25 MB 2025-02-14 11:46:50,383 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35154.56 MB 2025-02-14 11:46:50,383 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 11:46:50,383 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32857.03 MB 2025-02-14 11:46:50,384 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 11:46:50,384 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 11:46:50,384 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 11:46:50,384 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:46:50,384 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23181.36 MB 2025-02-14 11:46:50,384 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27312.75 MB 2025-02-14 11:46:50,384 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 11:46:50,384 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29492.25 MB 2025-02-14 11:46:50,384 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35154.56 MB 2025-02-14 11:46:50,384 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 11:46:50,384 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32857.03 MB 2025-02-14 11:46:50,555 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 11:46:50,555 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 11:46:50,555 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 11:46:50,555 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:46:50,555 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28846.29 MB 2025-02-14 11:46:50,556 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29613.29 MB 2025-02-14 11:46:50,556 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 11:46:50,556 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35154.56 MB 2025-02-14 11:46:50,556 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35569.80 MB 2025-02-14 11:46:50,556 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 11:46:50,556 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30321.08 MB 2025-02-14 11:46:50,574 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 11:46:50,574 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 11:46:50,574 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:46:50,574 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:46:50,574 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30026.18 MB 2025-02-14 11:46:50,574 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30255.26 MB 2025-02-14 11:46:50,574 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.08 MB 2025-02-14 11:46:50,574 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35569.80 MB 2025-02-14 11:46:50,574 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35569.80 MB 2025-02-14 11:46:50,574 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:46:50,574 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30484.56 MB 2025-02-14 11:46:50,575 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 11:46:50,575 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 11:46:50,575 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.88 seconds 2025-02-14 11:46:50,575 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:46:50,575 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17574.66 MB 2025-02-14 11:46:50,575 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30456.26 MB 2025-02-14 11:46:50,575 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12881.60 MB 2025-02-14 11:46:50,575 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57958.99 MB 2025-02-14 11:46:50,575 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35569.80 MB 2025-02-14 11:46:50,576 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22389.19 MB 2025-02-14 11:46:50,576 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30484.56 MB 2025-02-14 11:46:50,845 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 11:46:50,845 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 11:46:50,845 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 11:46:50,845 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:46:50,845 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30456.26 MB 2025-02-14 11:46:50,845 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22577.91 MB 2025-02-14 11:46:50,845 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7878.36 MB 2025-02-14 11:46:50,845 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35569.80 MB 2025-02-14 11:46:50,845 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35569.80 MB 2025-02-14 11:46:50,845 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:46:50,845 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32967.01 MB 2025-02-14 11:46:50,863 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8159, cut from 8161 2025-02-14 11:46:50,863 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 11:46:50,869 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 11:46:50,869 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 11:46:50,869 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:46:50,869 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:46:50,869 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22577.91 MB 2025-02-14 11:46:50,869 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31013.50 MB 2025-02-14 11:46:50,869 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8435.59 MB 2025-02-14 11:46:50,869 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35569.80 MB 2025-02-14 11:46:50,869 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39764.10 MB 2025-02-14 11:46:50,869 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4194.30 MB 2025-02-14 11:46:50,869 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31013.50 MB 2025-02-14 11:46:51,030 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7951] 2025-02-14 11:46:51,032 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:46:51,032 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 11:46:51,033 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:46:51,033 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 11:46:51,037 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 11:46:51,038 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:46:51,038 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 11:46:51,039 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 11:47:06,283 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:47:06,283 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 11:47:06,288 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 11:47:06,292 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:47:06,292 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 443, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 11:47:06,293 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:47:06,293 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 443, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 11:47:13,180 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 11:47:13,180 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 11:47:13,180 - resource_logging.py:150 - __exit__ - DEBUG - Time: 6.88 seconds 2025-02-14 11:47:13,180 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:47:13,180 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16055.60 MB 2025-02-14 11:47:13,180 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17624.27 MB 2025-02-14 11:47:13,180 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1568.67 MB 2025-02-14 11:47:13,180 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48152.71 MB 2025-02-14 11:47:13,180 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21309.16 MB 2025-02-14 11:47:13,180 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26843.55 MB 2025-02-14 11:47:13,180 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26433.75 MB 2025-02-14 11:47:13,211 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 11:47:13,211 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 11:47:13,211 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 11:47:13,211 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:47:13,211 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17624.27 MB 2025-02-14 11:47:13,211 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18081.92 MB 2025-02-14 11:47:13,211 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 457.65 MB 2025-02-14 11:47:13,211 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21309.16 MB 2025-02-14 11:47:13,211 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27963.42 MB 2025-02-14 11:47:13,211 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6654.26 MB 2025-02-14 11:47:13,211 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25001.61 MB 2025-02-14 11:47:15,136 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 11:47:15,136 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 11:47:15,136 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 11:47:15,136 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:47:15,136 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18081.92 MB 2025-02-14 11:47:15,136 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18612.76 MB 2025-02-14 11:47:15,136 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 11:47:15,136 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27963.42 MB 2025-02-14 11:47:15,136 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20921.19 MB 2025-02-14 11:47:15,136 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7042.24 MB 2025-02-14 11:47:15,136 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22593.13 MB 2025-02-14 11:47:15,150 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 11:47:15,150 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 11:47:15,150 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:47:15,150 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:47:15,150 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18612.76 MB 2025-02-14 11:47:15,150 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20502.29 MB 2025-02-14 11:47:15,150 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 11:47:15,150 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20921.19 MB 2025-02-14 11:47:15,150 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24224.20 MB 2025-02-14 11:47:15,150 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 11:47:15,150 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21919.72 MB 2025-02-14 11:47:15,382 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 11:47:15,382 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 11:47:15,382 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 11:47:15,382 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:47:15,382 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20502.29 MB 2025-02-14 11:47:15,382 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22744.15 MB 2025-02-14 11:47:15,382 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 11:47:15,382 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24224.20 MB 2025-02-14 11:47:15,382 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30830.23 MB 2025-02-14 11:47:15,382 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 11:47:15,382 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28288.43 MB 2025-02-14 11:47:15,383 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 11:47:15,383 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 11:47:15,383 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.25 seconds 2025-02-14 11:47:15,383 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:47:15,383 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18612.76 MB 2025-02-14 11:47:15,383 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22744.15 MB 2025-02-14 11:47:15,383 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 11:47:15,383 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20921.19 MB 2025-02-14 11:47:15,383 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30830.23 MB 2025-02-14 11:47:15,383 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9909.04 MB 2025-02-14 11:47:15,383 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28288.43 MB 2025-02-14 11:47:15,550 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 11:47:15,550 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 11:47:15,550 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 11:47:15,550 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:47:15,550 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24277.69 MB 2025-02-14 11:47:15,550 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25044.69 MB 2025-02-14 11:47:15,550 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 11:47:15,550 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30830.23 MB 2025-02-14 11:47:15,550 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31245.47 MB 2025-02-14 11:47:15,550 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 11:47:15,550 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25752.48 MB 2025-02-14 11:47:15,569 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 11:47:15,569 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 11:47:15,569 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:47:15,569 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:47:15,569 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25457.58 MB 2025-02-14 11:47:15,569 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25686.40 MB 2025-02-14 11:47:15,569 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.81 MB 2025-02-14 11:47:15,569 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31245.47 MB 2025-02-14 11:47:15,569 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31245.47 MB 2025-02-14 11:47:15,569 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:47:15,569 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25894.57 MB 2025-02-14 11:47:15,570 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 11:47:15,570 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 11:47:15,570 - resource_logging.py:150 - __exit__ - DEBUG - Time: 9.28 seconds 2025-02-14 11:47:15,570 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:47:15,570 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14512.15 MB 2025-02-14 11:47:15,570 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25887.47 MB 2025-02-14 11:47:15,570 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11375.32 MB 2025-02-14 11:47:15,570 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48152.71 MB 2025-02-14 11:47:15,570 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31245.47 MB 2025-02-14 11:47:15,570 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16907.24 MB 2025-02-14 11:47:15,570 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25894.57 MB 2025-02-14 11:47:15,837 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 11:47:15,837 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 11:47:15,837 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 11:47:15,837 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:47:15,837 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25887.47 MB 2025-02-14 11:47:15,837 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19516.54 MB 2025-02-14 11:47:15,838 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6370.93 MB 2025-02-14 11:47:15,838 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31245.47 MB 2025-02-14 11:47:15,838 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31245.47 MB 2025-02-14 11:47:15,838 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:47:15,838 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28399.14 MB 2025-02-14 11:47:15,856 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 11:47:15,856 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 11:47:15,862 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 11:47:15,862 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 11:47:15,862 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:47:15,862 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:47:15,862 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19516.54 MB 2025-02-14 11:47:15,862 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27955.57 MB 2025-02-14 11:47:15,862 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 11:47:15,862 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31245.47 MB 2025-02-14 11:47:15,862 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41735.42 MB 2025-02-14 11:47:15,862 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 11:47:15,862 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27955.57 MB 2025-02-14 11:47:16,024 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 11:47:16,026 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:47:16,026 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 11:47:16,027 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:47:16,027 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 11:47:16,031 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 11:47:16,032 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:47:16,033 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 11:47:16,033 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 11:48:10,661 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:48:10,662 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 11:48:10,666 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 11:48:10,670 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:48:10,670 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 305, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 11:48:10,671 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:48:10,671 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 305, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 11:48:15,342 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 11:48:15,342 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 11:48:15,342 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.67 seconds 2025-02-14 11:48:15,343 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:48:15,343 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15094.00 MB 2025-02-14 11:48:15,343 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16173.37 MB 2025-02-14 11:48:15,343 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1079.38 MB 2025-02-14 11:48:15,343 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54320.43 MB 2025-02-14 11:48:15,343 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18561.89 MB 2025-02-14 11:48:15,343 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35758.54 MB 2025-02-14 11:48:15,343 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25019.16 MB 2025-02-14 11:48:15,363 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 11:48:15,363 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 11:48:15,364 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:48:15,364 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:48:15,364 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16173.37 MB 2025-02-14 11:48:15,364 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16612.92 MB 2025-02-14 11:48:15,364 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 439.55 MB 2025-02-14 11:48:15,364 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18561.89 MB 2025-02-14 11:48:15,364 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22684.89 MB 2025-02-14 11:48:15,364 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4123.00 MB 2025-02-14 11:48:15,364 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20294.28 MB 2025-02-14 11:48:16,753 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 11:48:16,753 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 11:48:16,753 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.39 seconds 2025-02-14 11:48:16,753 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:48:16,753 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16612.92 MB 2025-02-14 11:48:16,753 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17001.76 MB 2025-02-14 11:48:16,753 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 388.84 MB 2025-02-14 11:48:16,753 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22684.89 MB 2025-02-14 11:48:16,753 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20623.39 MB 2025-02-14 11:48:16,753 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2061.50 MB 2025-02-14 11:48:16,753 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20954.27 MB 2025-02-14 11:48:16,763 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 11:48:16,764 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 11:48:16,764 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:48:16,764 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:48:16,764 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17001.76 MB 2025-02-14 11:48:16,764 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18386.74 MB 2025-02-14 11:48:16,764 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1384.97 MB 2025-02-14 11:48:16,764 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20623.39 MB 2025-02-14 11:48:16,764 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21315.45 MB 2025-02-14 11:48:16,764 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 692.06 MB 2025-02-14 11:48:16,764 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19425.01 MB 2025-02-14 11:48:16,922 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 11:48:16,922 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 11:48:16,922 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 11:48:16,922 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:48:16,922 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18386.74 MB 2025-02-14 11:48:16,922 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20029.70 MB 2025-02-14 11:48:16,922 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1642.96 MB 2025-02-14 11:48:16,922 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21315.45 MB 2025-02-14 11:48:16,922 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25813.84 MB 2025-02-14 11:48:16,922 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4498.39 MB 2025-02-14 11:48:16,922 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24094.41 MB 2025-02-14 11:48:16,923 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 11:48:16,923 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 11:48:16,923 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 11:48:16,923 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:48:16,923 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17001.76 MB 2025-02-14 11:48:16,923 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20029.70 MB 2025-02-14 11:48:16,923 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3027.94 MB 2025-02-14 11:48:16,923 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20623.39 MB 2025-02-14 11:48:16,923 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25813.84 MB 2025-02-14 11:48:16,923 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 11:48:16,923 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24094.41 MB 2025-02-14 11:48:17,048 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 11:48:17,048 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 11:48:17,048 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 11:48:17,048 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:48:17,048 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21153.41 MB 2025-02-14 11:48:17,048 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21715.24 MB 2025-02-14 11:48:17,048 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 561.83 MB 2025-02-14 11:48:17,048 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25813.84 MB 2025-02-14 11:48:17,048 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26115.83 MB 2025-02-14 11:48:17,048 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 301.99 MB 2025-02-14 11:48:17,048 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22233.70 MB 2025-02-14 11:48:17,063 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 11:48:17,063 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 11:48:17,063 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:48:17,063 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:48:17,063 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22017.69 MB 2025-02-14 11:48:17,063 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22235.77 MB 2025-02-14 11:48:17,063 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 218.08 MB 2025-02-14 11:48:17,063 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26115.83 MB 2025-02-14 11:48:17,063 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26115.83 MB 2025-02-14 11:48:17,063 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:48:17,063 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22320.54 MB 2025-02-14 11:48:17,064 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 11:48:17,064 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 11:48:17,064 - resource_logging.py:150 - __exit__ - DEBUG - Time: 6.39 seconds 2025-02-14 11:48:17,064 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:48:17,064 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14031.35 MB 2025-02-14 11:48:17,064 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22436.84 MB 2025-02-14 11:48:17,064 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8405.49 MB 2025-02-14 11:48:17,064 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54320.43 MB 2025-02-14 11:48:17,064 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26115.83 MB 2025-02-14 11:48:17,064 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28204.60 MB 2025-02-14 11:48:17,064 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22436.84 MB 2025-02-14 11:48:17,332 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 11:48:17,332 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 11:48:17,332 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 11:48:17,332 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:48:17,332 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22436.84 MB 2025-02-14 11:48:17,332 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25450.87 MB 2025-02-14 11:48:17,332 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-14 11:48:17,332 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26115.83 MB 2025-02-14 11:48:17,332 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26921.14 MB 2025-02-14 11:48:17,332 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 805.31 MB 2025-02-14 11:48:17,332 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25752.50 MB 2025-02-14 11:48:17,350 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 11:48:17,350 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 11:48:17,356 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 11:48:17,356 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 11:48:17,356 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:48:17,356 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:48:17,356 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18531.17 MB 2025-02-14 11:48:17,356 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26970.19 MB 2025-02-14 11:48:17,356 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 11:48:17,356 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26921.14 MB 2025-02-14 11:48:17,356 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37411.09 MB 2025-02-14 11:48:17,356 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 11:48:17,356 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26970.19 MB 2025-02-14 11:48:17,519 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 11:48:17,521 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:48:17,521 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 11:48:17,522 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:48:17,522 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 11:48:17,526 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 11:48:17,527 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:48:17,528 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 11:48:17,528 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 11:49:45,606 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:49:45,606 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 11:49:45,611 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 11:49:45,615 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:49:45,615 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1177, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 11:49:45,616 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:49:45,616 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1177, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 11:50:03,594 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 11:50:03,594 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 11:50:03,594 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.97 seconds 2025-02-14 11:50:03,594 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:50:03,594 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21170.23 MB 2025-02-14 11:50:03,594 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25335.57 MB 2025-02-14 11:50:03,594 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4165.34 MB 2025-02-14 11:50:03,594 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49996.10 MB 2025-02-14 11:50:03,594 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31566.33 MB 2025-02-14 11:50:03,594 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18429.77 MB 2025-02-14 11:50:03,594 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34265.48 MB 2025-02-14 11:50:03,670 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 11:50:03,670 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 11:50:03,670 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 11:50:03,670 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:50:03,670 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25335.57 MB 2025-02-14 11:50:03,670 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21896.71 MB 2025-02-14 11:50:03,670 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3438.86 MB 2025-02-14 11:50:03,670 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31566.33 MB 2025-02-14 11:50:03,670 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43593.50 MB 2025-02-14 11:50:03,670 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 12027.17 MB 2025-02-14 11:50:03,670 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37858.30 MB 2025-02-14 11:50:05,568 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 11:50:05,568 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 11:50:05,568 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.90 seconds 2025-02-14 11:50:05,568 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:50:05,568 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21896.71 MB 2025-02-14 11:50:05,568 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22427.55 MB 2025-02-14 11:50:05,568 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 11:50:05,568 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43593.50 MB 2025-02-14 11:50:05,568 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28814.87 MB 2025-02-14 11:50:05,568 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14778.63 MB 2025-02-14 11:50:05,568 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26406.88 MB 2025-02-14 11:50:05,581 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 11:50:05,581 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 11:50:05,581 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:50:05,581 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:50:05,581 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22427.55 MB 2025-02-14 11:50:05,581 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24317.08 MB 2025-02-14 11:50:05,581 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 11:50:05,581 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28814.87 MB 2025-02-14 11:50:05,581 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28814.87 MB 2025-02-14 11:50:05,581 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:50:05,581 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25734.51 MB 2025-02-14 11:50:05,799 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 11:50:05,799 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 11:50:05,799 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 11:50:05,799 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:50:05,799 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24317.08 MB 2025-02-14 11:50:05,799 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26558.94 MB 2025-02-14 11:50:05,799 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 11:50:05,799 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28814.87 MB 2025-02-14 11:50:05,799 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34477.18 MB 2025-02-14 11:50:05,799 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 11:50:05,799 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32103.22 MB 2025-02-14 11:50:05,800 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 11:50:05,800 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 11:50:05,800 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 11:50:05,800 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:50:05,800 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22427.55 MB 2025-02-14 11:50:05,800 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26558.94 MB 2025-02-14 11:50:05,800 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 11:50:05,800 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28814.87 MB 2025-02-14 11:50:05,800 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34477.18 MB 2025-02-14 11:50:05,800 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 11:50:05,800 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32103.22 MB 2025-02-14 11:50:05,976 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 11:50:05,976 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 11:50:05,976 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 11:50:05,976 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:50:05,976 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28092.48 MB 2025-02-14 11:50:05,976 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28859.48 MB 2025-02-14 11:50:05,976 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 11:50:05,976 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34477.18 MB 2025-02-14 11:50:05,976 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34890.32 MB 2025-02-14 11:50:05,976 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 11:50:05,976 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29567.27 MB 2025-02-14 11:50:05,996 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 11:50:05,996 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 11:50:05,996 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:50:05,996 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:50:05,996 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29272.37 MB 2025-02-14 11:50:05,996 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29500.94 MB 2025-02-14 11:50:05,996 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.57 MB 2025-02-14 11:50:05,996 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34890.32 MB 2025-02-14 11:50:05,996 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34890.32 MB 2025-02-14 11:50:05,996 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:50:05,996 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29742.25 MB 2025-02-14 11:50:05,997 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 11:50:05,997 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 11:50:05,997 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.38 seconds 2025-02-14 11:50:05,997 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:50:05,997 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17069.47 MB 2025-02-14 11:50:05,997 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29701.42 MB 2025-02-14 11:50:05,997 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12631.95 MB 2025-02-14 11:50:05,997 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49996.10 MB 2025-02-14 11:50:05,997 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34890.32 MB 2025-02-14 11:50:05,997 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15105.79 MB 2025-02-14 11:50:05,997 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29742.25 MB 2025-02-14 11:50:06,264 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 11:50:06,264 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 11:50:06,264 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 11:50:06,264 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:50:06,264 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29701.42 MB 2025-02-14 11:50:06,264 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22064.72 MB 2025-02-14 11:50:06,264 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7636.71 MB 2025-02-14 11:50:06,264 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34890.32 MB 2025-02-14 11:50:06,264 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34890.32 MB 2025-02-14 11:50:06,264 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:50:06,264 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32205.72 MB 2025-02-14 11:50:06,282 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8138, cut from 8140 2025-02-14 11:50:06,282 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 11:50:06,288 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 11:50:06,288 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 11:50:06,288 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:50:06,288 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:50:06,288 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22064.72 MB 2025-02-14 11:50:06,288 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30478.70 MB 2025-02-14 11:50:06,288 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8413.98 MB 2025-02-14 11:50:06,288 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34890.32 MB 2025-02-14 11:50:06,288 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43255.86 MB 2025-02-14 11:50:06,288 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8365.54 MB 2025-02-14 11:50:06,288 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30478.70 MB 2025-02-14 11:50:06,458 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7930] 2025-02-14 11:50:06,460 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:50:06,460 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 11:50:06,461 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:50:06,461 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 11:50:06,466 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 11:50:06,467 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:50:06,467 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 11:50:06,467 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 11:50:15,020 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:50:15,020 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 11:50:15,025 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 11:50:15,030 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:50:15,030 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1955, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 11:50:15,032 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:50:15,032 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1955, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 11:50:45,300 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 11:50:45,301 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 11:50:45,301 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.26 seconds 2025-02-14 11:50:45,301 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:50:45,301 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26591.46 MB 2025-02-14 11:50:45,301 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33510.09 MB 2025-02-14 11:50:45,301 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6918.64 MB 2025-02-14 11:50:45,301 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55803.12 MB 2025-02-14 11:50:45,301 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40558.92 MB 2025-02-14 11:50:45,301 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15244.20 MB 2025-02-14 11:50:45,301 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42405.42 MB 2025-02-14 11:50:45,419 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 11:50:45,419 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 11:50:45,419 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 11:50:45,419 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:50:45,419 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33510.09 MB 2025-02-14 11:50:45,419 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25941.29 MB 2025-02-14 11:50:45,419 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7568.81 MB 2025-02-14 11:50:45,419 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40558.92 MB 2025-02-14 11:50:45,419 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 61752.74 MB 2025-02-14 11:50:45,419 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 21193.82 MB 2025-02-14 11:50:45,419 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52390.03 MB 2025-02-14 11:50:47,358 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 11:50:47,358 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 11:50:47,358 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-14 11:50:47,358 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:50:47,358 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25941.29 MB 2025-02-14 11:50:47,358 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26472.13 MB 2025-02-14 11:50:47,359 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 11:50:47,359 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61752.74 MB 2025-02-14 11:50:47,359 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30872.17 MB 2025-02-14 11:50:47,359 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30880.56 MB 2025-02-14 11:50:47,359 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30452.50 MB 2025-02-14 11:50:47,372 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 11:50:47,372 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 11:50:47,372 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:50:47,372 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:50:47,372 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26472.13 MB 2025-02-14 11:50:47,372 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28361.66 MB 2025-02-14 11:50:47,372 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 11:50:47,372 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30872.17 MB 2025-02-14 11:50:47,372 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32759.61 MB 2025-02-14 11:50:47,372 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 11:50:47,373 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29779.09 MB 2025-02-14 11:50:47,583 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 11:50:47,583 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 11:50:47,583 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 11:50:47,583 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:50:47,583 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28361.66 MB 2025-02-14 11:50:47,583 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30603.52 MB 2025-02-14 11:50:47,583 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 11:50:47,583 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32759.61 MB 2025-02-14 11:50:47,583 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38893.78 MB 2025-02-14 11:50:47,583 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 11:50:47,583 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36147.80 MB 2025-02-14 11:50:47,584 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 11:50:47,584 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 11:50:47,584 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 11:50:47,584 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:50:47,584 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26472.13 MB 2025-02-14 11:50:47,584 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30603.52 MB 2025-02-14 11:50:47,584 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 11:50:47,584 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30872.17 MB 2025-02-14 11:50:47,584 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38893.78 MB 2025-02-14 11:50:47,584 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-14 11:50:47,584 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36147.80 MB 2025-02-14 11:50:47,752 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 11:50:47,752 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 11:50:47,752 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 11:50:47,752 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:50:47,752 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32137.06 MB 2025-02-14 11:50:47,752 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32904.06 MB 2025-02-14 11:50:47,752 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 11:50:47,752 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38893.78 MB 2025-02-14 11:50:47,752 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39309.02 MB 2025-02-14 11:50:47,752 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 11:50:47,752 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33611.85 MB 2025-02-14 11:50:47,771 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 11:50:47,771 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 11:50:47,771 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:50:47,771 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:50:47,771 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33316.95 MB 2025-02-14 11:50:47,771 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33545.60 MB 2025-02-14 11:50:47,771 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.65 MB 2025-02-14 11:50:47,771 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39309.02 MB 2025-02-14 11:50:47,771 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39309.02 MB 2025-02-14 11:50:47,771 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:50:47,771 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33746.33 MB 2025-02-14 11:50:47,772 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 11:50:47,772 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 11:50:47,772 - resource_logging.py:150 - __exit__ - DEBUG - Time: 32.74 seconds 2025-02-14 11:50:47,772 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:50:47,772 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19780.08 MB 2025-02-14 11:50:47,772 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33745.96 MB 2025-02-14 11:50:47,772 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13965.88 MB 2025-02-14 11:50:47,772 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55803.12 MB 2025-02-14 11:50:47,772 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39309.02 MB 2025-02-14 11:50:47,772 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16494.10 MB 2025-02-14 11:50:47,772 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33746.33 MB 2025-02-14 11:50:48,044 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 11:50:48,044 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 11:50:48,044 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 11:50:48,044 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:50:48,044 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33745.96 MB 2025-02-14 11:50:48,044 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24773.43 MB 2025-02-14 11:50:48,044 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8972.54 MB 2025-02-14 11:50:48,044 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39309.02 MB 2025-02-14 11:50:48,044 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39309.02 MB 2025-02-14 11:50:48,044 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:50:48,044 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36248.72 MB 2025-02-14 11:50:48,062 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8133, cut from 8135 2025-02-14 11:50:48,062 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 11:50:48,068 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 11:50:48,068 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 11:50:48,068 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:50:48,068 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:50:48,068 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24773.43 MB 2025-02-14 11:50:48,068 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33182.73 MB 2025-02-14 11:50:48,068 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8409.30 MB 2025-02-14 11:50:48,068 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39309.02 MB 2025-02-14 11:50:48,068 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47668.26 MB 2025-02-14 11:50:48,068 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8359.25 MB 2025-02-14 11:50:48,068 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33182.73 MB 2025-02-14 11:50:48,234 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7925] 2025-02-14 11:50:48,235 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:50:48,235 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 11:50:48,236 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:50:48,236 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 11:50:48,241 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 11:50:48,242 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:50:48,242 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 11:50:48,242 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 11:51:57,353 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:51:57,353 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 11:51:57,358 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 11:51:57,362 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:51:57,362 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 145, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 11:51:57,363 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:51:57,363 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 145, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 11:51:59,671 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 11:51:59,671 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 11:51:59,671 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.30 seconds 2025-02-14 11:51:59,671 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:51:59,671 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13979.09 MB 2025-02-14 11:51:59,671 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14492.24 MB 2025-02-14 11:51:59,671 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 513.15 MB 2025-02-14 11:51:59,671 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56027.51 MB 2025-02-14 11:51:59,671 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21562.92 MB 2025-02-14 11:51:59,671 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34464.60 MB 2025-02-14 11:51:59,671 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23450.96 MB 2025-02-14 11:51:59,685 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 11:51:59,685 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 11:51:59,685 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:51:59,686 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:51:59,686 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14492.24 MB 2025-02-14 11:51:59,686 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14740.86 MB 2025-02-14 11:51:59,686 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 248.62 MB 2025-02-14 11:51:59,686 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21562.92 MB 2025-02-14 11:51:59,686 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21562.92 MB 2025-02-14 11:51:59,686 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:51:59,686 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16532.52 MB 2025-02-14 11:52:00,403 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 11:52:00,403 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 11:52:00,403 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.71 seconds 2025-02-14 11:52:00,403 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:52:00,403 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14740.86 MB 2025-02-14 11:52:00,403 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14933.29 MB 2025-02-14 11:52:00,403 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 192.43 MB 2025-02-14 11:52:00,403 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21562.92 MB 2025-02-14 11:52:00,403 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21562.92 MB 2025-02-14 11:52:00,403 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:52:00,403 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18911.29 MB 2025-02-14 11:52:00,413 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 11:52:00,413 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 11:52:00,413 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 11:52:00,413 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:52:00,413 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14933.22 MB 2025-02-14 11:52:00,413 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15618.01 MB 2025-02-14 11:52:00,413 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 684.79 MB 2025-02-14 11:52:00,413 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21562.92 MB 2025-02-14 11:52:00,413 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21562.92 MB 2025-02-14 11:52:00,413 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:52:00,414 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16131.83 MB 2025-02-14 11:52:00,518 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 11:52:00,518 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 11:52:00,518 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 11:52:00,518 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:52:00,518 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15618.01 MB 2025-02-14 11:52:00,518 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16430.72 MB 2025-02-14 11:52:00,518 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 812.71 MB 2025-02-14 11:52:00,518 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21562.92 MB 2025-02-14 11:52:00,518 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21562.92 MB 2025-02-14 11:52:00,518 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:52:00,518 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18440.49 MB 2025-02-14 11:52:00,519 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 11:52:00,519 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 11:52:00,519 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 11:52:00,519 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:52:00,519 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14933.22 MB 2025-02-14 11:52:00,519 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16430.72 MB 2025-02-14 11:52:00,519 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1497.50 MB 2025-02-14 11:52:00,520 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21562.92 MB 2025-02-14 11:52:00,520 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21562.92 MB 2025-02-14 11:52:00,520 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:52:00,520 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18440.49 MB 2025-02-14 11:52:00,618 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 11:52:00,618 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 11:52:00,618 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 11:52:00,618 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:52:00,618 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16986.63 MB 2025-02-14 11:52:00,618 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17264.67 MB 2025-02-14 11:52:00,618 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 278.04 MB 2025-02-14 11:52:00,618 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21562.92 MB 2025-02-14 11:52:00,618 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21709.72 MB 2025-02-14 11:52:00,618 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 146.80 MB 2025-02-14 11:52:00,618 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17532.11 MB 2025-02-14 11:52:00,631 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 11:52:00,631 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 11:52:00,631 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:52:00,631 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:52:00,631 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17414.35 MB 2025-02-14 11:52:00,631 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17643.50 MB 2025-02-14 11:52:00,631 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.15 MB 2025-02-14 11:52:00,631 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21709.72 MB 2025-02-14 11:52:00,631 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21709.72 MB 2025-02-14 11:52:00,631 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:52:00,631 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17654.09 MB 2025-02-14 11:52:00,633 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 11:52:00,633 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 11:52:00,633 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.27 seconds 2025-02-14 11:52:00,633 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:52:00,633 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13473.90 MB 2025-02-14 11:52:00,633 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17844.48 MB 2025-02-14 11:52:00,633 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4370.58 MB 2025-02-14 11:52:00,634 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56027.51 MB 2025-02-14 11:52:00,634 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21709.72 MB 2025-02-14 11:52:00,634 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34317.80 MB 2025-02-14 11:52:00,634 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17844.48 MB 2025-02-14 11:52:00,914 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 11:52:00,914 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 11:52:00,914 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.28 seconds 2025-02-14 11:52:00,914 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:52:00,914 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17844.48 MB 2025-02-14 11:52:00,914 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17273.34 MB 2025-02-14 11:52:00,914 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -571.13 MB 2025-02-14 11:52:00,914 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21709.72 MB 2025-02-14 11:52:00,914 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21709.72 MB 2025-02-14 11:52:00,914 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:52:00,914 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18949.07 MB 2025-02-14 11:52:00,933 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8158, cut from 8160 2025-02-14 11:52:00,933 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 11:52:00,940 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 11:52:00,940 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 11:52:00,940 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:52:00,940 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:52:00,940 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17273.34 MB 2025-02-14 11:52:00,941 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25708.19 MB 2025-02-14 11:52:00,941 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8434.85 MB 2025-02-14 11:52:00,941 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21709.72 MB 2025-02-14 11:52:00,941 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30096.23 MB 2025-02-14 11:52:00,941 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8386.51 MB 2025-02-14 11:52:00,941 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25708.19 MB 2025-02-14 11:52:01,180 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7950] 2025-02-14 11:52:01,182 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:52:01,182 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 11:52:01,184 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:52:01,184 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 11:52:01,191 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 11:52:01,193 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:52:01,193 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 11:52:01,193 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 11:52:48,005 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:52:48,006 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 11:52:48,010 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 11:52:48,014 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:52:48,014 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1677, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 11:52:48,015 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:52:48,015 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1677, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 11:53:13,770 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 11:53:13,770 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 11:53:13,770 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.75 seconds 2025-02-14 11:53:13,770 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:53:13,770 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24654.31 MB 2025-02-14 11:53:13,770 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30589.25 MB 2025-02-14 11:53:13,770 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5934.94 MB 2025-02-14 11:53:13,770 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42674.95 MB 2025-02-14 11:53:13,770 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39602.62 MB 2025-02-14 11:53:13,770 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3072.33 MB 2025-02-14 11:53:13,770 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39562.31 MB 2025-02-14 11:53:13,868 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 11:53:13,868 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 11:53:13,868 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 11:53:13,868 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:53:13,868 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30589.25 MB 2025-02-14 11:53:13,868 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24496.05 MB 2025-02-14 11:53:13,868 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6093.20 MB 2025-02-14 11:53:13,868 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39602.62 MB 2025-02-14 11:53:13,868 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55662.61 MB 2025-02-14 11:53:13,868 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 16059.99 MB 2025-02-14 11:53:13,868 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47177.19 MB 2025-02-14 11:53:15,788 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 11:53:15,788 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 11:53:15,788 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 11:53:15,788 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:53:15,788 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24496.05 MB 2025-02-14 11:53:15,788 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25026.89 MB 2025-02-14 11:53:15,788 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 11:53:15,788 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55662.61 MB 2025-02-14 11:53:15,788 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30891.05 MB 2025-02-14 11:53:15,788 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24771.56 MB 2025-02-14 11:53:15,788 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29006.23 MB 2025-02-14 11:53:15,804 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 11:53:15,804 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 11:53:15,804 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:53:15,804 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:53:15,804 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25026.89 MB 2025-02-14 11:53:15,804 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26916.43 MB 2025-02-14 11:53:15,804 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 11:53:15,804 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30891.05 MB 2025-02-14 11:53:15,804 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31834.77 MB 2025-02-14 11:53:15,804 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 11:53:15,804 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28333.85 MB 2025-02-14 11:53:16,013 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 11:53:16,013 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 11:53:16,013 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 11:53:16,013 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:53:16,013 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26916.43 MB 2025-02-14 11:53:16,013 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29158.28 MB 2025-02-14 11:53:16,013 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 11:53:16,013 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31834.77 MB 2025-02-14 11:53:16,013 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37968.94 MB 2025-02-14 11:53:16,013 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 11:53:16,013 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34702.56 MB 2025-02-14 11:53:16,014 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 11:53:16,014 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 11:53:16,014 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 11:53:16,014 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:53:16,014 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25026.89 MB 2025-02-14 11:53:16,014 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29158.28 MB 2025-02-14 11:53:16,014 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 11:53:16,014 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30891.05 MB 2025-02-14 11:53:16,014 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37968.94 MB 2025-02-14 11:53:16,014 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7077.89 MB 2025-02-14 11:53:16,014 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34702.56 MB 2025-02-14 11:53:16,183 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 11:53:16,183 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 11:53:16,183 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 11:53:16,183 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:53:16,183 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30691.82 MB 2025-02-14 11:53:16,183 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31458.83 MB 2025-02-14 11:53:16,183 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 11:53:16,183 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37968.94 MB 2025-02-14 11:53:16,183 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38384.17 MB 2025-02-14 11:53:16,183 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 11:53:16,183 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32166.61 MB 2025-02-14 11:53:16,201 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 11:53:16,201 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 11:53:16,201 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:53:16,201 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:53:16,201 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31871.71 MB 2025-02-14 11:53:16,202 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32100.79 MB 2025-02-14 11:53:16,202 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.07 MB 2025-02-14 11:53:16,202 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38384.17 MB 2025-02-14 11:53:16,202 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38384.17 MB 2025-02-14 11:53:16,202 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:53:16,202 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32309.27 MB 2025-02-14 11:53:16,203 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 11:53:16,203 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 11:53:16,203 - resource_logging.py:150 - __exit__ - DEBUG - Time: 28.19 seconds 2025-02-14 11:53:16,203 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:53:16,203 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18811.51 MB 2025-02-14 11:53:16,203 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32301.29 MB 2025-02-14 11:53:16,203 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13489.79 MB 2025-02-14 11:53:16,203 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42674.95 MB 2025-02-14 11:53:16,203 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38384.17 MB 2025-02-14 11:53:16,203 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4290.77 MB 2025-02-14 11:53:16,203 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32309.27 MB 2025-02-14 11:53:16,472 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 11:53:16,472 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 11:53:16,472 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 11:53:16,472 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:53:16,472 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32301.29 MB 2025-02-14 11:53:16,472 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23807.84 MB 2025-02-14 11:53:16,472 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8493.46 MB 2025-02-14 11:53:16,472 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38384.17 MB 2025-02-14 11:53:16,472 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38384.17 MB 2025-02-14 11:53:16,472 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:53:16,472 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34806.60 MB 2025-02-14 11:53:16,490 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8139, cut from 8141 2025-02-14 11:53:16,490 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 11:53:16,496 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 11:53:16,496 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 11:53:16,496 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:53:16,496 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:53:16,496 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23807.84 MB 2025-02-14 11:53:16,496 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32222.79 MB 2025-02-14 11:53:16,496 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8414.95 MB 2025-02-14 11:53:16,496 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38384.17 MB 2025-02-14 11:53:16,496 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46751.81 MB 2025-02-14 11:53:16,496 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8367.64 MB 2025-02-14 11:53:16,496 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32222.79 MB 2025-02-14 11:53:16,658 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7931] 2025-02-14 11:53:16,660 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:53:16,660 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 11:53:16,661 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:53:16,661 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 11:53:16,666 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 11:53:16,667 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:53:16,667 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 11:53:16,667 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 11:53:26,940 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:53:26,940 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 11:53:26,945 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 11:53:26,949 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:53:26,949 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1011, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 11:53:26,950 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:53:26,950 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1011, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 11:53:42,646 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 11:53:42,646 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 11:53:42,646 - resource_logging.py:150 - __exit__ - DEBUG - Time: 15.69 seconds 2025-02-14 11:53:42,646 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:53:42,646 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20013.52 MB 2025-02-14 11:53:42,646 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23591.39 MB 2025-02-14 11:53:42,646 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3577.87 MB 2025-02-14 11:53:42,646 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55119.45 MB 2025-02-14 11:53:42,646 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28856.81 MB 2025-02-14 11:53:42,646 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26262.63 MB 2025-02-14 11:53:42,646 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32430.10 MB 2025-02-14 11:53:42,711 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 11:53:42,711 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 11:53:42,711 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 11:53:42,711 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:53:42,711 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23591.39 MB 2025-02-14 11:53:42,711 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21033.72 MB 2025-02-14 11:53:42,711 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2557.66 MB 2025-02-14 11:53:42,711 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28856.81 MB 2025-02-14 11:53:42,711 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40533.75 MB 2025-02-14 11:53:42,711 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 11676.94 MB 2025-02-14 11:53:42,711 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34710.80 MB 2025-02-14 11:53:44,648 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 11:53:44,648 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 11:53:44,648 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 11:53:44,648 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:53:44,648 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21033.72 MB 2025-02-14 11:53:44,648 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21564.57 MB 2025-02-14 11:53:44,648 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 11:53:44,648 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40533.75 MB 2025-02-14 11:53:44,648 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26692.55 MB 2025-02-14 11:53:44,648 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13841.20 MB 2025-02-14 11:53:44,648 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25543.90 MB 2025-02-14 11:53:44,663 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 11:53:44,663 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 11:53:44,663 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:53:44,663 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:53:44,663 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21564.57 MB 2025-02-14 11:53:44,663 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23454.10 MB 2025-02-14 11:53:44,663 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 11:53:44,663 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26692.55 MB 2025-02-14 11:53:44,663 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28579.99 MB 2025-02-14 11:53:44,663 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 11:53:44,663 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24871.53 MB 2025-02-14 11:53:44,874 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 11:53:44,874 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 11:53:44,874 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 11:53:44,874 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:53:44,874 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23454.10 MB 2025-02-14 11:53:44,874 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25695.96 MB 2025-02-14 11:53:44,874 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 11:53:44,874 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28579.99 MB 2025-02-14 11:53:44,874 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34242.30 MB 2025-02-14 11:53:44,874 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 11:53:44,874 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31240.24 MB 2025-02-14 11:53:44,875 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 11:53:44,875 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 11:53:44,875 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 11:53:44,875 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:53:44,875 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21564.57 MB 2025-02-14 11:53:44,875 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25695.96 MB 2025-02-14 11:53:44,875 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 11:53:44,875 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26692.55 MB 2025-02-14 11:53:44,875 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34242.30 MB 2025-02-14 11:53:44,875 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-14 11:53:44,875 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31240.24 MB 2025-02-14 11:53:45,042 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 11:53:45,042 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 11:53:45,042 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 11:53:45,042 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:53:45,042 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27229.50 MB 2025-02-14 11:53:45,042 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27996.50 MB 2025-02-14 11:53:45,042 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 11:53:45,042 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34242.30 MB 2025-02-14 11:53:45,042 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34655.44 MB 2025-02-14 11:53:45,042 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 11:53:45,042 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28704.29 MB 2025-02-14 11:53:45,060 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 11:53:45,060 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 11:53:45,060 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:53:45,060 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:53:45,060 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28409.39 MB 2025-02-14 11:53:45,060 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28638.20 MB 2025-02-14 11:53:45,060 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.81 MB 2025-02-14 11:53:45,060 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34655.44 MB 2025-02-14 11:53:45,060 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34655.44 MB 2025-02-14 11:53:45,060 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:53:45,060 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28859.45 MB 2025-02-14 11:53:45,062 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 11:53:45,062 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 11:53:45,062 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.11 seconds 2025-02-14 11:53:45,062 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:53:45,062 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16491.11 MB 2025-02-14 11:53:45,062 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28838.93 MB 2025-02-14 11:53:45,062 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12347.82 MB 2025-02-14 11:53:45,062 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55119.45 MB 2025-02-14 11:53:45,062 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34655.44 MB 2025-02-14 11:53:45,062 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20464.01 MB 2025-02-14 11:53:45,062 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28859.45 MB 2025-02-14 11:53:45,330 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 11:53:45,330 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 11:53:45,330 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 11:53:45,330 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:53:45,330 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28838.93 MB 2025-02-14 11:53:45,330 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21490.17 MB 2025-02-14 11:53:45,330 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7348.76 MB 2025-02-14 11:53:45,330 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34655.44 MB 2025-02-14 11:53:45,330 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34655.44 MB 2025-02-14 11:53:45,330 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:53:45,330 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31346.30 MB 2025-02-14 11:53:45,348 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8148, cut from 8150 2025-02-14 11:53:45,348 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 11:53:45,355 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 11:53:45,355 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 11:53:45,355 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:53:45,355 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:53:45,355 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21490.17 MB 2025-02-14 11:53:45,355 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29915.12 MB 2025-02-14 11:53:45,355 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8424.95 MB 2025-02-14 11:53:45,355 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34655.44 MB 2025-02-14 11:53:45,355 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45126.52 MB 2025-02-14 11:53:45,355 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10471.08 MB 2025-02-14 11:53:45,355 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29915.12 MB 2025-02-14 11:53:45,521 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7940] 2025-02-14 11:53:45,523 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:53:45,523 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 11:53:45,524 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:53:45,524 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 11:53:45,528 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 11:53:45,529 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:53:45,529 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 11:53:45,530 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 11:54:11,625 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:54:11,625 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 11:54:11,630 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 11:54:11,634 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:54:11,634 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 186, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 11:54:11,635 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:54:11,635 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 186, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 11:54:14,529 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 11:54:14,529 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 11:54:14,529 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.89 seconds 2025-02-14 11:54:14,529 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:54:14,529 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14264.78 MB 2025-02-14 11:54:14,529 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14923.03 MB 2025-02-14 11:54:14,529 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 658.24 MB 2025-02-14 11:54:14,529 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53502.54 MB 2025-02-14 11:54:14,529 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 16909.34 MB 2025-02-14 11:54:14,529 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -36593.21 MB 2025-02-14 11:54:14,529 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23736.96 MB 2025-02-14 11:54:14,543 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 11:54:14,543 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 11:54:14,543 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:54:14,543 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:54:14,543 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14923.03 MB 2025-02-14 11:54:14,543 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15242.21 MB 2025-02-14 11:54:14,543 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 319.18 MB 2025-02-14 11:54:14,543 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 16909.34 MB 2025-02-14 11:54:14,543 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18547.21 MB 2025-02-14 11:54:14,543 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1637.88 MB 2025-02-14 11:54:14,543 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17571.57 MB 2025-02-14 11:54:15,451 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 11:54:15,451 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 11:54:15,451 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.91 seconds 2025-02-14 11:54:15,451 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:54:15,451 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15242.21 MB 2025-02-14 11:54:15,451 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15489.05 MB 2025-02-14 11:54:15,451 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 246.84 MB 2025-02-14 11:54:15,451 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18547.21 MB 2025-02-14 11:54:15,451 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17932.75 MB 2025-02-14 11:54:15,451 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -614.47 MB 2025-02-14 11:54:15,451 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19413.68 MB 2025-02-14 11:54:15,459 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 11:54:15,459 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 11:54:15,459 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:54:15,459 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:54:15,459 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15488.98 MB 2025-02-14 11:54:15,459 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16367.40 MB 2025-02-14 11:54:15,459 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 878.42 MB 2025-02-14 11:54:15,459 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17932.75 MB 2025-02-14 11:54:15,459 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18373.15 MB 2025-02-14 11:54:15,459 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 440.40 MB 2025-02-14 11:54:15,459 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17026.51 MB 2025-02-14 11:54:15,559 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 11:54:15,559 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 11:54:15,559 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 11:54:15,559 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:54:15,559 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16367.40 MB 2025-02-14 11:54:15,559 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17409.90 MB 2025-02-14 11:54:15,559 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1042.50 MB 2025-02-14 11:54:15,559 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18373.15 MB 2025-02-14 11:54:15,559 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21015.56 MB 2025-02-14 11:54:15,559 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2642.41 MB 2025-02-14 11:54:15,559 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19987.96 MB 2025-02-14 11:54:15,560 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 11:54:15,560 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 11:54:15,560 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 11:54:15,560 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:54:15,560 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15488.98 MB 2025-02-14 11:54:15,560 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17409.90 MB 2025-02-14 11:54:15,560 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1920.92 MB 2025-02-14 11:54:15,560 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17932.75 MB 2025-02-14 11:54:15,560 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21015.56 MB 2025-02-14 11:54:15,560 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3082.81 MB 2025-02-14 11:54:15,560 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19987.96 MB 2025-02-14 11:54:15,639 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 11:54:15,639 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 11:54:15,640 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 11:54:15,640 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:54:15,640 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18123.00 MB 2025-02-14 11:54:15,640 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18479.65 MB 2025-02-14 11:54:15,640 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 356.66 MB 2025-02-14 11:54:15,640 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21015.56 MB 2025-02-14 11:54:15,640 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21206.40 MB 2025-02-14 11:54:15,640 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 190.84 MB 2025-02-14 11:54:15,640 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18813.49 MB 2025-02-14 11:54:15,650 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 11:54:15,650 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 11:54:15,650 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:54:15,650 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:54:15,650 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18671.65 MB 2025-02-14 11:54:15,650 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18875.40 MB 2025-02-14 11:54:15,650 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 203.75 MB 2025-02-14 11:54:15,650 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21206.40 MB 2025-02-14 11:54:15,650 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21210.60 MB 2025-02-14 11:54:15,650 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-14 11:54:15,650 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18903.16 MB 2025-02-14 11:54:15,652 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 11:54:15,652 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 11:54:15,652 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.02 seconds 2025-02-14 11:54:15,652 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:54:15,652 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13616.74 MB 2025-02-14 11:54:15,652 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19076.40 MB 2025-02-14 11:54:15,652 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5459.66 MB 2025-02-14 11:54:15,652 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53502.54 MB 2025-02-14 11:54:15,652 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21210.60 MB 2025-02-14 11:54:15,652 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32291.95 MB 2025-02-14 11:54:15,652 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19076.40 MB 2025-02-14 11:54:15,919 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 11:54:15,919 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 11:54:15,919 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 11:54:15,919 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:54:15,919 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19076.40 MB 2025-02-14 11:54:15,919 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17609.54 MB 2025-02-14 11:54:15,919 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1466.86 MB 2025-02-14 11:54:15,919 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21210.60 MB 2025-02-14 11:54:15,919 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21210.60 MB 2025-02-14 11:54:15,919 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:54:15,919 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19076.41 MB 2025-02-14 11:54:15,937 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8159, cut from 8161 2025-02-14 11:54:15,937 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 11:54:15,943 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 11:54:15,943 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 11:54:15,943 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:54:15,943 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:54:15,943 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17609.54 MB 2025-02-14 11:54:15,943 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26045.13 MB 2025-02-14 11:54:15,943 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8435.59 MB 2025-02-14 11:54:15,943 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21210.60 MB 2025-02-14 11:54:15,943 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31696.36 MB 2025-02-14 11:54:15,943 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10485.76 MB 2025-02-14 11:54:15,943 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26045.13 MB 2025-02-14 11:54:16,106 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7951] 2025-02-14 11:54:16,107 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:54:16,107 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 11:54:16,108 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:54:16,108 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 11:54:16,113 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 11:54:16,114 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:54:16,114 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 11:54:16,114 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 11:55:07,991 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:55:07,991 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 11:55:07,996 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 11:55:08,000 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:55:08,000 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 608, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 11:55:08,001 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:55:08,001 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 608, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 11:55:17,327 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 11:55:17,327 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 11:55:17,327 - resource_logging.py:150 - __exit__ - DEBUG - Time: 9.32 seconds 2025-02-14 11:55:17,327 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:55:17,327 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17205.35 MB 2025-02-14 11:55:17,327 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19357.03 MB 2025-02-14 11:55:17,327 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2151.68 MB 2025-02-14 11:55:17,328 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40084.96 MB 2025-02-14 11:55:17,328 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23198.70 MB 2025-02-14 11:55:17,328 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16886.27 MB 2025-02-14 11:55:17,328 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28262.16 MB 2025-02-14 11:55:17,360 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 11:55:17,360 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 11:55:17,360 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 11:55:17,360 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:55:17,360 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19357.03 MB 2025-02-14 11:55:17,360 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18609.23 MB 2025-02-14 11:55:17,360 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -747.80 MB 2025-02-14 11:55:17,360 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23198.70 MB 2025-02-14 11:55:17,360 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26304.58 MB 2025-02-14 11:55:17,360 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3105.88 MB 2025-02-14 11:55:17,360 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24374.82 MB 2025-02-14 11:55:19,040 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 11:55:19,040 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 11:55:19,040 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.68 seconds 2025-02-14 11:55:19,040 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:55:19,040 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18609.23 MB 2025-02-14 11:55:19,040 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19077.69 MB 2025-02-14 11:55:19,040 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 468.47 MB 2025-02-14 11:55:19,040 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26304.58 MB 2025-02-14 11:55:19,040 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24448.60 MB 2025-02-14 11:55:19,040 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1855.98 MB 2025-02-14 11:55:19,040 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23034.47 MB 2025-02-14 11:55:19,052 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 11:55:19,052 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 11:55:19,052 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:55:19,052 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:55:19,052 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19077.69 MB 2025-02-14 11:55:19,052 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20745.45 MB 2025-02-14 11:55:19,052 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1667.76 MB 2025-02-14 11:55:19,052 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24448.60 MB 2025-02-14 11:55:19,053 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24448.60 MB 2025-02-14 11:55:19,053 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:55:19,053 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21996.34 MB 2025-02-14 11:55:19,242 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 11:55:19,242 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 11:55:19,242 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.19 seconds 2025-02-14 11:55:19,242 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:55:19,242 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20745.45 MB 2025-02-14 11:55:19,242 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22724.82 MB 2025-02-14 11:55:19,242 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1979.36 MB 2025-02-14 11:55:19,242 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24448.60 MB 2025-02-14 11:55:19,242 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29456.60 MB 2025-02-14 11:55:19,242 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5008.00 MB 2025-02-14 11:55:19,242 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27618.56 MB 2025-02-14 11:55:19,243 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 11:55:19,243 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 11:55:19,243 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 11:55:19,243 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:55:19,243 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19077.69 MB 2025-02-14 11:55:19,243 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22724.82 MB 2025-02-14 11:55:19,243 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3647.12 MB 2025-02-14 11:55:19,243 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24448.60 MB 2025-02-14 11:55:19,243 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29456.60 MB 2025-02-14 11:55:19,243 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5008.00 MB 2025-02-14 11:55:19,243 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27618.56 MB 2025-02-14 11:55:19,392 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 11:55:19,393 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 11:55:19,393 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-14 11:55:19,393 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:55:19,393 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24078.17 MB 2025-02-14 11:55:19,393 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24755.97 MB 2025-02-14 11:55:19,393 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 677.80 MB 2025-02-14 11:55:19,393 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29456.60 MB 2025-02-14 11:55:19,393 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29823.60 MB 2025-02-14 11:55:19,393 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 367.00 MB 2025-02-14 11:55:19,393 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25380.59 MB 2025-02-14 11:55:19,410 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 11:55:19,410 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 11:55:19,410 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:55:19,410 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:55:19,410 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25120.34 MB 2025-02-14 11:55:19,410 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25356.78 MB 2025-02-14 11:55:19,410 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 236.44 MB 2025-02-14 11:55:19,410 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29823.60 MB 2025-02-14 11:55:19,410 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29825.70 MB 2025-02-14 11:55:19,410 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-14 11:55:19,410 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25489.97 MB 2025-02-14 11:55:19,411 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 11:55:19,411 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 11:55:19,411 - resource_logging.py:150 - __exit__ - DEBUG - Time: 11.41 seconds 2025-02-14 11:55:19,411 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:55:19,411 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15087.03 MB 2025-02-14 11:55:19,411 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25557.85 MB 2025-02-14 11:55:19,411 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 10470.83 MB 2025-02-14 11:55:19,411 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40084.96 MB 2025-02-14 11:55:19,411 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29825.70 MB 2025-02-14 11:55:19,411 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10259.27 MB 2025-02-14 11:55:19,411 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25557.85 MB 2025-02-14 11:55:19,678 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 11:55:19,678 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 11:55:19,678 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 11:55:19,678 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:55:19,678 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25557.85 MB 2025-02-14 11:55:19,678 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28571.89 MB 2025-02-14 11:55:19,678 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-14 11:55:19,678 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29825.70 MB 2025-02-14 11:55:19,678 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29825.70 MB 2025-02-14 11:55:19,678 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:55:19,678 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28873.25 MB 2025-02-14 11:55:19,734 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 11:55:19,735 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 11:55:19,743 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 11:55:19,743 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 11:55:19,743 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 11:55:19,744 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:55:19,744 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19870.53 MB 2025-02-14 11:55:19,744 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28309.55 MB 2025-02-14 11:55:19,744 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 11:55:19,744 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29825.70 MB 2025-02-14 11:55:19,744 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38216.40 MB 2025-02-14 11:55:19,744 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 11:55:19,744 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28309.55 MB 2025-02-14 11:55:19,907 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 11:55:19,909 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:55:19,909 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 11:55:19,910 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:55:19,910 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 11:55:19,914 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 11:55:19,915 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:55:19,915 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 11:55:19,916 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 11:56:15,758 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:56:15,759 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 11:56:15,764 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 11:56:15,768 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:56:15,768 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1324, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 11:56:15,769 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:56:15,769 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1324, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 11:56:36,201 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 11:56:36,201 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 11:56:36,201 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.42 seconds 2025-02-14 11:56:36,201 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:56:36,201 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22194.55 MB 2025-02-14 11:56:36,201 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26880.11 MB 2025-02-14 11:56:36,201 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4685.56 MB 2025-02-14 11:56:36,201 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50801.41 MB 2025-02-14 11:56:36,201 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38373.69 MB 2025-02-14 11:56:36,201 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12427.72 MB 2025-02-14 11:56:36,201 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35742.79 MB 2025-02-14 11:56:36,258 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 11:56:36,258 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 11:56:36,258 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 11:56:36,258 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:56:36,258 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26880.11 MB 2025-02-14 11:56:36,258 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22660.91 MB 2025-02-14 11:56:36,258 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4219.20 MB 2025-02-14 11:56:36,258 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38373.69 MB 2025-02-14 11:56:36,258 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39789.26 MB 2025-02-14 11:56:36,258 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1415.58 MB 2025-02-14 11:56:36,258 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34230.47 MB 2025-02-14 11:56:38,167 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 11:56:38,167 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 11:56:38,167 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 11:56:38,167 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:56:38,167 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22660.91 MB 2025-02-14 11:56:38,167 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23191.76 MB 2025-02-14 11:56:38,167 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 11:56:38,167 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39789.26 MB 2025-02-14 11:56:38,167 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30907.83 MB 2025-02-14 11:56:38,167 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8881.44 MB 2025-02-14 11:56:38,167 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27171.09 MB 2025-02-14 11:56:38,180 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 11:56:38,180 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 11:56:38,180 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:56:38,180 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:56:38,180 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23191.76 MB 2025-02-14 11:56:38,180 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25081.29 MB 2025-02-14 11:56:38,180 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 11:56:38,180 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30907.83 MB 2025-02-14 11:56:38,180 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30907.83 MB 2025-02-14 11:56:38,180 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:56:38,180 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26498.72 MB 2025-02-14 11:56:38,394 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 11:56:38,394 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 11:56:38,394 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 11:56:38,394 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:56:38,394 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25081.29 MB 2025-02-14 11:56:38,394 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27323.15 MB 2025-02-14 11:56:38,394 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 11:56:38,394 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30907.83 MB 2025-02-14 11:56:38,394 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35626.42 MB 2025-02-14 11:56:38,394 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 11:56:38,394 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32867.43 MB 2025-02-14 11:56:38,395 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 11:56:38,395 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 11:56:38,395 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 11:56:38,395 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:56:38,395 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23191.76 MB 2025-02-14 11:56:38,395 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27323.15 MB 2025-02-14 11:56:38,395 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 11:56:38,395 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30907.83 MB 2025-02-14 11:56:38,395 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35626.42 MB 2025-02-14 11:56:38,395 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 11:56:38,395 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32867.43 MB 2025-02-14 11:56:38,573 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 11:56:38,573 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 11:56:38,573 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 11:56:38,573 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:56:38,574 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28856.69 MB 2025-02-14 11:56:38,574 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29623.69 MB 2025-02-14 11:56:38,574 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 11:56:38,574 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35626.42 MB 2025-02-14 11:56:38,574 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36041.65 MB 2025-02-14 11:56:38,574 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 11:56:38,574 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30331.48 MB 2025-02-14 11:56:38,593 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 11:56:38,593 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 11:56:38,593 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:56:38,593 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:56:38,593 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30036.58 MB 2025-02-14 11:56:38,593 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30271.70 MB 2025-02-14 11:56:38,593 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 235.12 MB 2025-02-14 11:56:38,593 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36041.65 MB 2025-02-14 11:56:38,593 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36043.75 MB 2025-02-14 11:56:38,593 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-14 11:56:38,593 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30415.92 MB 2025-02-14 11:56:38,594 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 11:56:38,594 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 11:56:38,594 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.82 seconds 2025-02-14 11:56:38,594 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:56:38,594 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17581.63 MB 2025-02-14 11:56:38,594 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30472.77 MB 2025-02-14 11:56:38,594 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12891.14 MB 2025-02-14 11:56:38,594 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50801.41 MB 2025-02-14 11:56:38,594 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36043.75 MB 2025-02-14 11:56:38,594 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14757.66 MB 2025-02-14 11:56:38,594 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30472.77 MB 2025-02-14 11:56:38,863 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 11:56:38,863 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 11:56:38,863 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 11:56:38,863 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:56:38,863 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30472.77 MB 2025-02-14 11:56:38,863 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22586.02 MB 2025-02-14 11:56:38,863 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7886.75 MB 2025-02-14 11:56:38,863 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36043.75 MB 2025-02-14 11:56:38,863 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36043.75 MB 2025-02-14 11:56:38,863 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:56:38,863 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32984.44 MB 2025-02-14 11:56:38,881 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 11:56:38,881 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 11:56:38,887 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 11:56:38,887 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 11:56:38,887 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:56:38,887 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:56:38,887 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22586.02 MB 2025-02-14 11:56:38,887 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31025.04 MB 2025-02-14 11:56:38,887 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 11:56:38,887 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36043.75 MB 2025-02-14 11:56:38,887 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40240.15 MB 2025-02-14 11:56:38,887 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4196.40 MB 2025-02-14 11:56:38,887 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31025.04 MB 2025-02-14 11:56:39,053 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 11:56:39,054 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:56:39,054 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 11:56:39,055 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:56:39,055 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 11:56:39,060 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 11:56:39,061 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:56:39,061 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 11:56:39,061 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 11:57:34,462 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:57:34,463 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 11:57:34,468 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 11:57:34,472 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:57:34,472 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1533, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 11:57:34,473 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:57:34,473 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1533, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 11:57:58,079 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 11:57:58,079 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 11:57:58,079 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.60 seconds 2025-02-14 11:57:58,079 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:57:58,079 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23650.90 MB 2025-02-14 11:57:58,079 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29076.23 MB 2025-02-14 11:57:58,079 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5425.33 MB 2025-02-14 11:57:58,079 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52825.16 MB 2025-02-14 11:57:58,079 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39116.08 MB 2025-02-14 11:57:58,079 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13709.08 MB 2025-02-14 11:57:58,079 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37879.41 MB 2025-02-14 11:57:58,167 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 11:57:58,167 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 11:57:58,167 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 11:57:58,167 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:57:58,167 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29076.23 MB 2025-02-14 11:57:58,167 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23747.44 MB 2025-02-14 11:57:58,167 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5328.79 MB 2025-02-14 11:57:58,167 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39116.08 MB 2025-02-14 11:57:58,167 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49222.25 MB 2025-02-14 11:57:58,167 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10106.18 MB 2025-02-14 11:57:58,167 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44131.60 MB 2025-02-14 11:58:00,086 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 11:58:00,086 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 11:58:00,086 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 11:58:00,086 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:58:00,086 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23747.44 MB 2025-02-14 11:58:00,086 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24278.28 MB 2025-02-14 11:58:00,086 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 11:58:00,086 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49222.25 MB 2025-02-14 11:58:00,086 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33690.75 MB 2025-02-14 11:58:00,086 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15531.51 MB 2025-02-14 11:58:00,086 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28257.61 MB 2025-02-14 11:58:00,099 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 11:58:00,100 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 11:58:00,100 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:58:00,100 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:58:00,100 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24278.28 MB 2025-02-14 11:58:00,100 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26167.82 MB 2025-02-14 11:58:00,100 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 11:58:00,100 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33690.75 MB 2025-02-14 11:58:00,100 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33690.75 MB 2025-02-14 11:58:00,100 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:58:00,100 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27585.24 MB 2025-02-14 11:58:00,313 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 11:58:00,313 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 11:58:00,313 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 11:58:00,313 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:58:00,313 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26167.82 MB 2025-02-14 11:58:00,313 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28409.67 MB 2025-02-14 11:58:00,313 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 11:58:00,313 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33690.75 MB 2025-02-14 11:58:00,313 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37465.62 MB 2025-02-14 11:58:00,313 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 11:58:00,313 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33953.95 MB 2025-02-14 11:58:00,314 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 11:58:00,314 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 11:58:00,314 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 11:58:00,314 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:58:00,314 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24278.28 MB 2025-02-14 11:58:00,314 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28409.67 MB 2025-02-14 11:58:00,314 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 11:58:00,314 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33690.75 MB 2025-02-14 11:58:00,314 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37465.62 MB 2025-02-14 11:58:00,314 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 11:58:00,314 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33953.95 MB 2025-02-14 11:58:00,487 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 11:58:00,487 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 11:58:00,487 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 11:58:00,487 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:58:00,487 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29943.21 MB 2025-02-14 11:58:00,487 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30710.22 MB 2025-02-14 11:58:00,487 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 11:58:00,487 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37465.62 MB 2025-02-14 11:58:00,487 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37876.66 MB 2025-02-14 11:58:00,487 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 411.04 MB 2025-02-14 11:58:00,487 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31418.00 MB 2025-02-14 11:58:00,506 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 11:58:00,506 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 11:58:00,506 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:58:00,506 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:58:00,506 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31123.10 MB 2025-02-14 11:58:00,506 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31351.67 MB 2025-02-14 11:58:00,506 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.57 MB 2025-02-14 11:58:00,506 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37876.66 MB 2025-02-14 11:58:00,506 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37876.66 MB 2025-02-14 11:58:00,506 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:58:00,506 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31544.91 MB 2025-02-14 11:58:00,507 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 11:58:00,507 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 11:58:00,507 - resource_logging.py:150 - __exit__ - DEBUG - Time: 26.03 seconds 2025-02-14 11:58:00,507 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:58:00,507 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18309.80 MB 2025-02-14 11:58:00,507 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31552.15 MB 2025-02-14 11:58:00,507 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13242.35 MB 2025-02-14 11:58:00,508 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52825.16 MB 2025-02-14 11:58:00,508 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37876.66 MB 2025-02-14 11:58:00,508 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14948.50 MB 2025-02-14 11:58:00,508 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31552.15 MB 2025-02-14 11:58:00,776 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 11:58:00,776 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 11:58:00,776 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 11:58:00,776 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:58:00,776 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31552.15 MB 2025-02-14 11:58:00,776 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23305.05 MB 2025-02-14 11:58:00,776 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8247.11 MB 2025-02-14 11:58:00,776 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37876.66 MB 2025-02-14 11:58:00,776 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37876.66 MB 2025-02-14 11:58:00,776 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:58:00,776 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34056.45 MB 2025-02-14 11:58:00,793 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8138, cut from 8140 2025-02-14 11:58:00,794 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 11:58:00,800 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 11:58:00,800 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 11:58:00,800 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:58:00,800 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:58:00,800 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23305.05 MB 2025-02-14 11:58:00,800 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31719.03 MB 2025-02-14 11:58:00,800 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8413.98 MB 2025-02-14 11:58:00,800 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37876.66 MB 2025-02-14 11:58:00,800 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42058.38 MB 2025-02-14 11:58:00,800 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4181.72 MB 2025-02-14 11:58:00,800 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31719.03 MB 2025-02-14 11:58:00,965 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7930] 2025-02-14 11:58:00,966 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:58:00,966 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 11:58:00,967 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:58:00,967 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 11:58:00,972 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 11:58:00,973 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:58:00,973 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 11:58:00,973 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 11:58:58,149 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:58:58,149 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 11:58:58,154 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 11:58:58,158 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:58:58,158 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1276, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 11:58:58,159 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:58:58,159 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1276, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 11:59:17,673 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 11:59:17,673 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 11:59:17,673 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.51 seconds 2025-02-14 11:59:17,673 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:59:17,673 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21860.08 MB 2025-02-14 11:59:17,673 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26375.77 MB 2025-02-14 11:59:17,673 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4515.69 MB 2025-02-14 11:59:17,673 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54605.64 MB 2025-02-14 11:59:17,673 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38157.68 MB 2025-02-14 11:59:17,673 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16447.96 MB 2025-02-14 11:59:17,673 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35181.82 MB 2025-02-14 11:59:17,747 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 11:59:17,747 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 11:59:17,747 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 11:59:17,747 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:59:17,747 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26375.77 MB 2025-02-14 11:59:17,747 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22411.38 MB 2025-02-14 11:59:17,747 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3964.39 MB 2025-02-14 11:59:17,747 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38157.68 MB 2025-02-14 11:59:17,747 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46988.79 MB 2025-02-14 11:59:17,747 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8831.11 MB 2025-02-14 11:59:17,747 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39696.13 MB 2025-02-14 11:59:19,656 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 11:59:19,656 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 11:59:19,656 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 11:59:19,656 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:59:19,656 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22411.38 MB 2025-02-14 11:59:19,656 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22942.22 MB 2025-02-14 11:59:19,656 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 11:59:19,656 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46988.79 MB 2025-02-14 11:59:19,656 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33640.42 MB 2025-02-14 11:59:19,656 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13348.37 MB 2025-02-14 11:59:19,656 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26921.55 MB 2025-02-14 11:59:19,669 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 11:59:19,669 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 11:59:19,669 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 11:59:19,669 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:59:19,670 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22942.22 MB 2025-02-14 11:59:19,670 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24831.75 MB 2025-02-14 11:59:19,670 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 11:59:19,670 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33640.42 MB 2025-02-14 11:59:19,670 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33640.42 MB 2025-02-14 11:59:19,670 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:59:19,670 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26249.18 MB 2025-02-14 11:59:19,881 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 11:59:19,881 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 11:59:19,881 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 11:59:19,881 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:59:19,881 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24831.75 MB 2025-02-14 11:59:19,881 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27073.61 MB 2025-02-14 11:59:19,881 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 11:59:19,881 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33640.42 MB 2025-02-14 11:59:19,881 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35527.85 MB 2025-02-14 11:59:19,882 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 11:59:19,882 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32617.89 MB 2025-02-14 11:59:19,882 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 11:59:19,882 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 11:59:19,882 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 11:59:19,882 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:59:19,882 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22942.22 MB 2025-02-14 11:59:19,882 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27073.61 MB 2025-02-14 11:59:19,882 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 11:59:19,882 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33640.42 MB 2025-02-14 11:59:19,882 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35527.85 MB 2025-02-14 11:59:19,882 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 11:59:19,882 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32617.89 MB 2025-02-14 11:59:20,051 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 11:59:20,051 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 11:59:20,051 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 11:59:20,051 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:59:20,051 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28607.15 MB 2025-02-14 11:59:20,051 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29374.15 MB 2025-02-14 11:59:20,051 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 11:59:20,051 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35527.85 MB 2025-02-14 11:59:20,051 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35940.99 MB 2025-02-14 11:59:20,051 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 11:59:20,051 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30081.94 MB 2025-02-14 11:59:20,070 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 11:59:20,070 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 11:59:20,070 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:59:20,070 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:59:20,070 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29787.04 MB 2025-02-14 11:59:20,070 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30016.00 MB 2025-02-14 11:59:20,070 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.96 MB 2025-02-14 11:59:20,070 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35940.99 MB 2025-02-14 11:59:20,070 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35940.99 MB 2025-02-14 11:59:20,070 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:59:20,070 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30252.54 MB 2025-02-14 11:59:20,071 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 11:59:20,071 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 11:59:20,071 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.91 seconds 2025-02-14 11:59:20,071 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:59:20,071 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17414.39 MB 2025-02-14 11:59:20,071 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30216.88 MB 2025-02-14 11:59:20,071 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12802.49 MB 2025-02-14 11:59:20,071 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54605.64 MB 2025-02-14 11:59:20,071 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35940.99 MB 2025-02-14 11:59:20,071 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18664.65 MB 2025-02-14 11:59:20,071 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30252.54 MB 2025-02-14 11:59:20,339 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 11:59:20,339 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 11:59:20,339 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 11:59:20,339 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:59:20,339 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30216.88 MB 2025-02-14 11:59:20,339 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22416.13 MB 2025-02-14 11:59:20,339 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7800.75 MB 2025-02-14 11:59:20,339 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35940.99 MB 2025-02-14 11:59:20,339 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35940.99 MB 2025-02-14 11:59:20,339 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 11:59:20,339 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32726.09 MB 2025-02-14 11:59:20,357 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8154, cut from 8156 2025-02-14 11:59:20,357 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 11:59:20,363 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 11:59:20,363 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 11:59:20,363 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 11:59:20,363 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 11:59:20,363 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22416.13 MB 2025-02-14 11:59:20,363 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30846.80 MB 2025-02-14 11:59:20,364 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8430.68 MB 2025-02-14 11:59:20,364 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35940.99 MB 2025-02-14 11:59:20,364 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44323.31 MB 2025-02-14 11:59:20,364 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8382.32 MB 2025-02-14 11:59:20,364 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30846.80 MB 2025-02-14 11:59:20,529 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7946] 2025-02-14 11:59:20,530 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:59:20,530 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 11:59:20,531 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:59:20,531 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 11:59:20,536 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 11:59:20,537 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 11:59:20,537 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 11:59:20,537 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 12:00:20,297 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:00:20,297 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 12:00:20,302 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 12:00:20,305 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:00:20,306 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1462, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 12:00:20,306 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:00:20,307 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1462, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 12:00:42,699 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 12:00:42,699 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 12:00:42,699 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.39 seconds 2025-02-14 12:00:42,699 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:00:42,699 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23156.16 MB 2025-02-14 12:00:42,699 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28330.09 MB 2025-02-14 12:00:42,699 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5173.94 MB 2025-02-14 12:00:42,699 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56895.73 MB 2025-02-14 12:00:42,699 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38814.09 MB 2025-02-14 12:00:42,699 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18081.64 MB 2025-02-14 12:00:42,699 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37157.38 MB 2025-02-14 12:00:42,777 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 12:00:42,777 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 12:00:42,777 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 12:00:42,777 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:00:42,777 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28330.09 MB 2025-02-14 12:00:42,777 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23378.33 MB 2025-02-14 12:00:42,777 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4951.76 MB 2025-02-14 12:00:42,777 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38814.09 MB 2025-02-14 12:00:42,777 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47026.54 MB 2025-02-14 12:00:42,777 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8212.45 MB 2025-02-14 12:00:42,777 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40787.18 MB 2025-02-14 12:00:44,691 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 12:00:44,691 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 12:00:44,691 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 12:00:44,691 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:00:44,691 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23378.33 MB 2025-02-14 12:00:44,691 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23909.17 MB 2025-02-14 12:00:44,691 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 12:00:44,691 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47026.54 MB 2025-02-14 12:00:44,691 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29456.60 MB 2025-02-14 12:00:44,691 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17569.94 MB 2025-02-14 12:00:44,691 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27888.51 MB 2025-02-14 12:00:44,707 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 12:00:44,707 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 12:00:44,707 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:00:44,707 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:00:44,707 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23909.17 MB 2025-02-14 12:00:44,707 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25798.71 MB 2025-02-14 12:00:44,707 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 12:00:44,707 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29456.60 MB 2025-02-14 12:00:44,707 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30400.32 MB 2025-02-14 12:00:44,707 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 12:00:44,707 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27216.14 MB 2025-02-14 12:00:44,924 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 12:00:44,924 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 12:00:44,924 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 12:00:44,924 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:00:44,924 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25798.71 MB 2025-02-14 12:00:44,924 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28040.56 MB 2025-02-14 12:00:44,924 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 12:00:44,924 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30400.32 MB 2025-02-14 12:00:44,924 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36062.63 MB 2025-02-14 12:00:44,924 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 12:00:44,924 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33584.85 MB 2025-02-14 12:00:44,925 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 12:00:44,925 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 12:00:44,925 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 12:00:44,925 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:00:44,925 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23909.17 MB 2025-02-14 12:00:44,925 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28040.56 MB 2025-02-14 12:00:44,925 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 12:00:44,925 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29456.60 MB 2025-02-14 12:00:44,925 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36062.63 MB 2025-02-14 12:00:44,925 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 12:00:44,925 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33584.85 MB 2025-02-14 12:00:45,096 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 12:00:45,096 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 12:00:45,096 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 12:00:45,096 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:00:45,096 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29574.11 MB 2025-02-14 12:00:45,096 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30341.11 MB 2025-02-14 12:00:45,096 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 12:00:45,096 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36062.63 MB 2025-02-14 12:00:45,096 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36477.86 MB 2025-02-14 12:00:45,096 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 12:00:45,096 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31048.90 MB 2025-02-14 12:00:45,115 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 12:00:45,115 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 12:00:45,115 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:00:45,115 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:00:45,115 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30754.00 MB 2025-02-14 12:00:45,115 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30982.44 MB 2025-02-14 12:00:45,115 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.44 MB 2025-02-14 12:00:45,115 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36477.86 MB 2025-02-14 12:00:45,115 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36477.86 MB 2025-02-14 12:00:45,115 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:00:45,115 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31207.88 MB 2025-02-14 12:00:45,116 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 12:00:45,116 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 12:00:45,116 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.81 seconds 2025-02-14 12:00:45,116 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:00:45,116 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18062.43 MB 2025-02-14 12:00:45,116 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31183.44 MB 2025-02-14 12:00:45,116 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13121.01 MB 2025-02-14 12:00:45,116 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56895.73 MB 2025-02-14 12:00:45,116 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36477.86 MB 2025-02-14 12:00:45,116 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20417.87 MB 2025-02-14 12:00:45,116 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31207.88 MB 2025-02-14 12:00:45,383 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 12:00:45,383 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 12:00:45,383 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 12:00:45,383 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:00:45,383 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31183.44 MB 2025-02-14 12:00:45,383 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23065.68 MB 2025-02-14 12:00:45,383 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8117.76 MB 2025-02-14 12:00:45,383 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36477.86 MB 2025-02-14 12:00:45,383 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36477.86 MB 2025-02-14 12:00:45,383 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:00:45,383 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33694.19 MB 2025-02-14 12:00:45,401 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8159, cut from 8161 2025-02-14 12:00:45,402 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 12:00:45,408 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 12:00:45,408 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 12:00:45,408 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:00:45,408 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:00:45,408 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23065.68 MB 2025-02-14 12:00:45,408 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31501.27 MB 2025-02-14 12:00:45,408 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8435.59 MB 2025-02-14 12:00:45,408 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36477.86 MB 2025-02-14 12:00:45,408 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44866.47 MB 2025-02-14 12:00:45,408 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8388.61 MB 2025-02-14 12:00:45,408 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31501.27 MB 2025-02-14 12:00:45,571 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7951] 2025-02-14 12:00:45,572 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:00:45,572 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 12:00:45,573 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:00:45,573 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 12:00:45,578 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 12:00:45,579 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:00:45,579 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 12:00:45,579 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 12:00:54,954 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:00:54,954 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 12:00:54,963 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 12:00:54,969 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:00:54,969 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1370, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 12:00:54,971 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:00:54,971 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1370, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 12:01:16,350 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 12:01:16,350 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 12:01:16,350 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.37 seconds 2025-02-14 12:01:16,350 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:01:16,350 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22515.09 MB 2025-02-14 12:01:16,350 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27363.70 MB 2025-02-14 12:01:16,350 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4848.62 MB 2025-02-14 12:01:16,350 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53255.08 MB 2025-02-14 12:01:16,350 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38501.61 MB 2025-02-14 12:01:16,350 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14753.46 MB 2025-02-14 12:01:16,350 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36289.81 MB 2025-02-14 12:01:16,426 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 12:01:16,426 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 12:01:16,426 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 12:01:16,426 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:01:16,426 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27363.70 MB 2025-02-14 12:01:16,426 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22900.05 MB 2025-02-14 12:01:16,426 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4463.65 MB 2025-02-14 12:01:16,426 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38501.61 MB 2025-02-14 12:01:16,426 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46959.43 MB 2025-02-14 12:01:16,426 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8457.81 MB 2025-02-14 12:01:16,426 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40118.03 MB 2025-02-14 12:01:18,352 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 12:01:18,352 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 12:01:18,352 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 12:01:18,352 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:01:18,352 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22900.05 MB 2025-02-14 12:01:18,352 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23430.89 MB 2025-02-14 12:01:18,352 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 12:01:18,352 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46959.43 MB 2025-02-14 12:01:18,352 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33653.00 MB 2025-02-14 12:01:18,353 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13306.43 MB 2025-02-14 12:01:18,353 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27410.23 MB 2025-02-14 12:01:18,366 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 12:01:18,366 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 12:01:18,366 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:01:18,366 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:01:18,366 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23430.89 MB 2025-02-14 12:01:18,366 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25320.43 MB 2025-02-14 12:01:18,366 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 12:01:18,366 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33653.00 MB 2025-02-14 12:01:18,366 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33653.00 MB 2025-02-14 12:01:18,366 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:01:18,366 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26737.86 MB 2025-02-14 12:01:18,578 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 12:01:18,578 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 12:01:18,578 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 12:01:18,578 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:01:18,578 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25320.43 MB 2025-02-14 12:01:18,578 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27562.28 MB 2025-02-14 12:01:18,578 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 12:01:18,578 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33653.00 MB 2025-02-14 12:01:18,578 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37427.87 MB 2025-02-14 12:01:18,578 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 12:01:18,578 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33106.57 MB 2025-02-14 12:01:18,578 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 12:01:18,578 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 12:01:18,578 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 12:01:18,578 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:01:18,578 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23430.89 MB 2025-02-14 12:01:18,579 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27562.28 MB 2025-02-14 12:01:18,579 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 12:01:18,579 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33653.00 MB 2025-02-14 12:01:18,579 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37427.87 MB 2025-02-14 12:01:18,579 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 12:01:18,579 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33106.57 MB 2025-02-14 12:01:18,750 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 12:01:18,750 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 12:01:18,750 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 12:01:18,750 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:01:18,750 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29095.83 MB 2025-02-14 12:01:18,750 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29862.83 MB 2025-02-14 12:01:18,750 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 12:01:18,750 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37427.87 MB 2025-02-14 12:01:18,750 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37841.01 MB 2025-02-14 12:01:18,750 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 12:01:18,750 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30570.62 MB 2025-02-14 12:01:18,769 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 12:01:18,769 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 12:01:18,769 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:01:18,769 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:01:18,769 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30275.72 MB 2025-02-14 12:01:18,769 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30504.75 MB 2025-02-14 12:01:18,769 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.03 MB 2025-02-14 12:01:18,769 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37841.01 MB 2025-02-14 12:01:18,769 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37841.01 MB 2025-02-14 12:01:18,769 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:01:18,769 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30729.74 MB 2025-02-14 12:01:18,770 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 12:01:18,770 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 12:01:18,770 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.80 seconds 2025-02-14 12:01:18,770 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:01:18,770 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17741.90 MB 2025-02-14 12:01:18,770 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30705.70 MB 2025-02-14 12:01:18,770 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12963.81 MB 2025-02-14 12:01:18,770 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53255.08 MB 2025-02-14 12:01:18,770 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37841.01 MB 2025-02-14 12:01:18,770 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15414.07 MB 2025-02-14 12:01:18,770 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30729.74 MB 2025-02-14 12:01:19,040 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 12:01:19,040 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 12:01:19,040 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 12:01:19,040 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:01:19,040 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30705.70 MB 2025-02-14 12:01:19,040 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22744.38 MB 2025-02-14 12:01:19,040 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7961.32 MB 2025-02-14 12:01:19,040 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37841.01 MB 2025-02-14 12:01:19,040 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37841.01 MB 2025-02-14 12:01:19,040 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:01:19,040 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33215.83 MB 2025-02-14 12:01:19,058 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8157, cut from 8159 2025-02-14 12:01:19,058 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 12:01:19,064 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 12:01:19,064 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 12:01:19,064 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:01:19,064 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:01:19,064 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22744.38 MB 2025-02-14 12:01:19,064 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31179.00 MB 2025-02-14 12:01:19,064 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8434.62 MB 2025-02-14 12:01:19,064 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37841.01 MB 2025-02-14 12:01:19,064 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46225.42 MB 2025-02-14 12:01:19,064 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8384.41 MB 2025-02-14 12:01:19,064 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31179.00 MB 2025-02-14 12:01:19,229 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7949] 2025-02-14 12:01:19,231 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:01:19,231 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 12:01:19,232 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:01:19,232 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 12:01:19,236 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 12:01:19,237 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:01:19,237 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 12:01:19,238 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 12:02:10,751 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:02:10,751 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 12:02:10,756 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 12:02:10,759 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:02:10,759 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 168, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 12:02:10,760 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:02:10,760 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 168, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 12:02:13,392 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 12:02:13,392 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 12:02:13,392 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.63 seconds 2025-02-14 12:02:13,392 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:02:13,392 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14139.36 MB 2025-02-14 12:02:13,392 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14733.90 MB 2025-02-14 12:02:13,392 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 594.54 MB 2025-02-14 12:02:13,392 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54609.84 MB 2025-02-14 12:02:13,392 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 16911.43 MB 2025-02-14 12:02:13,392 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -37698.40 MB 2025-02-14 12:02:13,392 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23611.53 MB 2025-02-14 12:02:13,404 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 12:02:13,404 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 12:02:13,404 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:02:13,404 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:02:13,404 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14733.90 MB 2025-02-14 12:02:13,404 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14979.82 MB 2025-02-14 12:02:13,404 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 245.92 MB 2025-02-14 12:02:13,404 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 16911.43 MB 2025-02-14 12:02:13,404 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18339.59 MB 2025-02-14 12:02:13,404 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1428.16 MB 2025-02-14 12:02:13,404 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17037.74 MB 2025-02-14 12:02:14,184 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 12:02:14,184 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 12:02:14,184 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.78 seconds 2025-02-14 12:02:14,184 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:02:14,184 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14979.82 MB 2025-02-14 12:02:14,184 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15194.81 MB 2025-02-14 12:02:14,184 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 214.99 MB 2025-02-14 12:02:14,184 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18339.59 MB 2025-02-14 12:02:14,184 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17764.97 MB 2025-02-14 12:02:14,184 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -574.62 MB 2025-02-14 12:02:14,184 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19152.33 MB 2025-02-14 12:02:14,192 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 12:02:14,192 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 12:02:14,192 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 12:02:14,192 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:02:14,192 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15194.74 MB 2025-02-14 12:02:14,192 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15959.82 MB 2025-02-14 12:02:14,192 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 765.08 MB 2025-02-14 12:02:14,192 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17764.97 MB 2025-02-14 12:02:14,192 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17764.97 MB 2025-02-14 12:02:14,192 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:02:14,192 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16533.88 MB 2025-02-14 12:02:14,283 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 12:02:14,283 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 12:02:14,283 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 12:02:14,283 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:02:14,283 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15959.82 MB 2025-02-14 12:02:14,283 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16867.81 MB 2025-02-14 12:02:14,283 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 907.99 MB 2025-02-14 12:02:14,283 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17764.97 MB 2025-02-14 12:02:14,283 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20069.74 MB 2025-02-14 12:02:14,283 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2304.77 MB 2025-02-14 12:02:14,283 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19113.20 MB 2025-02-14 12:02:14,284 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 12:02:14,284 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 12:02:14,284 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 12:02:14,284 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:02:14,284 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15194.74 MB 2025-02-14 12:02:14,284 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16867.81 MB 2025-02-14 12:02:14,284 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1673.07 MB 2025-02-14 12:02:14,284 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17764.97 MB 2025-02-14 12:02:14,284 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20069.74 MB 2025-02-14 12:02:14,284 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2304.77 MB 2025-02-14 12:02:14,284 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19113.20 MB 2025-02-14 12:02:14,355 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 12:02:14,355 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 12:02:14,355 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 12:02:14,355 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:02:14,355 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17488.89 MB 2025-02-14 12:02:14,355 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17799.53 MB 2025-02-14 12:02:14,355 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 310.64 MB 2025-02-14 12:02:14,355 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20069.74 MB 2025-02-14 12:02:14,355 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20233.32 MB 2025-02-14 12:02:14,355 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 163.58 MB 2025-02-14 12:02:14,355 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18096.63 MB 2025-02-14 12:02:14,365 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 12:02:14,365 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 12:02:14,365 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:02:14,365 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:02:14,365 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17966.76 MB 2025-02-14 12:02:14,365 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18171.42 MB 2025-02-14 12:02:14,365 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 204.66 MB 2025-02-14 12:02:14,365 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20233.32 MB 2025-02-14 12:02:14,365 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20237.52 MB 2025-02-14 12:02:14,365 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-14 12:02:14,365 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18195.58 MB 2025-02-14 12:02:14,366 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 12:02:14,366 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 12:02:14,367 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.60 seconds 2025-02-14 12:02:14,367 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:02:14,367 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13554.03 MB 2025-02-14 12:02:14,367 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18372.37 MB 2025-02-14 12:02:14,367 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4818.34 MB 2025-02-14 12:02:14,367 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54609.84 MB 2025-02-14 12:02:14,367 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20237.52 MB 2025-02-14 12:02:14,367 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34372.32 MB 2025-02-14 12:02:14,367 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18372.37 MB 2025-02-14 12:02:14,635 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 12:02:14,635 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 12:02:14,635 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 12:02:14,635 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:02:14,635 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18372.37 MB 2025-02-14 12:02:14,635 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17433.06 MB 2025-02-14 12:02:14,635 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -939.31 MB 2025-02-14 12:02:14,635 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20237.52 MB 2025-02-14 12:02:14,635 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20237.52 MB 2025-02-14 12:02:14,635 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:02:14,635 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19075.21 MB 2025-02-14 12:02:14,653 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8157, cut from 8159 2025-02-14 12:02:14,653 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 12:02:14,659 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 12:02:14,659 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 12:02:14,659 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:02:14,659 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:02:14,659 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17433.06 MB 2025-02-14 12:02:14,659 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25867.68 MB 2025-02-14 12:02:14,659 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8434.62 MB 2025-02-14 12:02:14,659 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20237.52 MB 2025-02-14 12:02:14,659 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30719.08 MB 2025-02-14 12:02:14,659 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10481.57 MB 2025-02-14 12:02:14,659 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25867.68 MB 2025-02-14 12:02:14,828 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7949] 2025-02-14 12:02:14,829 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:02:14,829 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 12:02:14,830 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:02:14,830 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 12:02:14,835 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 12:02:14,836 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:02:14,836 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 12:02:14,837 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 12:02:50,655 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:02:50,656 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 12:02:50,661 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 12:02:50,665 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:02:50,665 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1277, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 12:02:50,666 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:02:50,666 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1277, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 12:03:10,370 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 12:03:10,370 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 12:03:10,371 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.70 seconds 2025-02-14 12:03:10,371 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:03:10,371 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21867.05 MB 2025-02-14 12:03:10,371 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26386.41 MB 2025-02-14 12:03:10,371 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4519.36 MB 2025-02-14 12:03:10,371 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39103.50 MB 2025-02-14 12:03:10,371 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38197.53 MB 2025-02-14 12:03:10,371 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -905.97 MB 2025-02-14 12:03:10,371 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35188.79 MB 2025-02-14 12:03:10,445 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 12:03:10,446 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 12:03:10,446 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 12:03:10,446 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:03:10,446 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26386.41 MB 2025-02-14 12:03:10,446 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22416.58 MB 2025-02-14 12:03:10,446 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3969.83 MB 2025-02-14 12:03:10,446 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38197.53 MB 2025-02-14 12:03:10,446 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47001.37 MB 2025-02-14 12:03:10,446 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8803.84 MB 2025-02-14 12:03:10,446 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39666.60 MB 2025-02-14 12:03:12,362 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 12:03:12,362 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 12:03:12,362 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 12:03:12,362 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:03:12,362 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22416.58 MB 2025-02-14 12:03:12,362 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22947.42 MB 2025-02-14 12:03:12,362 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 12:03:12,362 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47001.37 MB 2025-02-14 12:03:12,362 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29485.96 MB 2025-02-14 12:03:12,362 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17515.41 MB 2025-02-14 12:03:12,362 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26926.75 MB 2025-02-14 12:03:12,376 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 12:03:12,376 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 12:03:12,376 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:03:12,376 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:03:12,376 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22947.42 MB 2025-02-14 12:03:12,376 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24836.95 MB 2025-02-14 12:03:12,376 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 12:03:12,376 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29485.96 MB 2025-02-14 12:03:12,376 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29485.96 MB 2025-02-14 12:03:12,376 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:03:12,376 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26254.38 MB 2025-02-14 12:03:12,591 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 12:03:12,591 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 12:03:12,591 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 12:03:12,591 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:03:12,591 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24836.95 MB 2025-02-14 12:03:12,591 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27078.81 MB 2025-02-14 12:03:12,591 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 12:03:12,591 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29485.96 MB 2025-02-14 12:03:12,591 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35148.27 MB 2025-02-14 12:03:12,591 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 12:03:12,591 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32623.09 MB 2025-02-14 12:03:12,592 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 12:03:12,592 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 12:03:12,592 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 12:03:12,592 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:03:12,592 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22947.42 MB 2025-02-14 12:03:12,592 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27078.81 MB 2025-02-14 12:03:12,592 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 12:03:12,592 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29485.96 MB 2025-02-14 12:03:12,592 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35148.27 MB 2025-02-14 12:03:12,592 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 12:03:12,592 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32623.09 MB 2025-02-14 12:03:12,796 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 12:03:12,796 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 12:03:12,796 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 12:03:12,796 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:03:12,796 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28612.35 MB 2025-02-14 12:03:12,796 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29379.35 MB 2025-02-14 12:03:12,796 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 12:03:12,796 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35148.27 MB 2025-02-14 12:03:12,796 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35563.50 MB 2025-02-14 12:03:12,796 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 12:03:12,796 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30087.14 MB 2025-02-14 12:03:12,815 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 12:03:12,815 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 12:03:12,815 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:03:12,815 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:03:12,815 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29792.24 MB 2025-02-14 12:03:12,815 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30020.13 MB 2025-02-14 12:03:12,815 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.89 MB 2025-02-14 12:03:12,815 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35563.50 MB 2025-02-14 12:03:12,815 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35563.50 MB 2025-02-14 12:03:12,815 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:03:12,815 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30253.45 MB 2025-02-14 12:03:12,816 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 12:03:12,816 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 12:03:12,816 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.15 seconds 2025-02-14 12:03:12,816 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:03:12,816 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17417.88 MB 2025-02-14 12:03:12,816 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30220.49 MB 2025-02-14 12:03:12,816 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12802.62 MB 2025-02-14 12:03:12,816 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39103.50 MB 2025-02-14 12:03:12,816 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35563.50 MB 2025-02-14 12:03:12,816 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3539.99 MB 2025-02-14 12:03:12,816 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30253.45 MB 2025-02-14 12:03:13,085 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 12:03:13,085 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 12:03:13,085 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 12:03:13,085 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:03:13,085 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30220.49 MB 2025-02-14 12:03:13,085 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22411.22 MB 2025-02-14 12:03:13,085 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7809.27 MB 2025-02-14 12:03:13,085 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35563.50 MB 2025-02-14 12:03:13,085 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35563.50 MB 2025-02-14 12:03:13,085 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:03:13,085 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32723.25 MB 2025-02-14 12:03:13,103 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8133, cut from 8135 2025-02-14 12:03:13,103 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 12:03:13,109 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 12:03:13,109 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 12:03:13,109 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:03:13,109 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:03:13,109 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22411.22 MB 2025-02-14 12:03:13,109 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30819.99 MB 2025-02-14 12:03:13,109 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8408.77 MB 2025-02-14 12:03:13,109 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35563.50 MB 2025-02-14 12:03:13,109 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39743.13 MB 2025-02-14 12:03:13,109 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4179.62 MB 2025-02-14 12:03:13,109 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30819.99 MB 2025-02-14 12:03:13,273 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7925] 2025-02-14 12:03:13,274 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:03:13,274 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 12:03:13,275 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:03:13,275 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 12:03:13,280 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 12:03:13,281 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:03:13,281 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 12:03:13,281 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 12:03:24,692 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:03:24,692 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 12:03:24,697 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 12:03:24,701 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:03:24,701 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1001, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 12:03:24,702 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:03:24,702 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1001, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 12:03:40,437 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 12:03:40,437 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 12:03:40,437 - resource_logging.py:150 - __exit__ - DEBUG - Time: 15.73 seconds 2025-02-14 12:03:40,437 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:03:40,437 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19943.83 MB 2025-02-14 12:03:40,437 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23486.32 MB 2025-02-14 12:03:40,437 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3542.48 MB 2025-02-14 12:03:40,437 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48102.38 MB 2025-02-14 12:03:40,437 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28825.35 MB 2025-02-14 12:03:40,437 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19277.02 MB 2025-02-14 12:03:40,437 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32360.41 MB 2025-02-14 12:03:40,501 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 12:03:40,501 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 12:03:40,501 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 12:03:40,501 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:03:40,501 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23486.32 MB 2025-02-14 12:03:40,501 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20981.74 MB 2025-02-14 12:03:40,501 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2504.58 MB 2025-02-14 12:03:40,501 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28825.35 MB 2025-02-14 12:03:40,501 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39472.59 MB 2025-02-14 12:03:40,501 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10647.24 MB 2025-02-14 12:03:40,501 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34227.35 MB 2025-02-14 12:03:42,422 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 12:03:42,423 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 12:03:42,423 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 12:03:42,423 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:03:42,423 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20981.74 MB 2025-02-14 12:03:42,423 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21512.58 MB 2025-02-14 12:03:42,423 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 12:03:42,423 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39472.59 MB 2025-02-14 12:03:42,423 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26696.74 MB 2025-02-14 12:03:42,423 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12775.85 MB 2025-02-14 12:03:42,423 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25491.91 MB 2025-02-14 12:03:42,436 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 12:03:42,436 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 12:03:42,436 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:03:42,436 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:03:42,436 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21512.58 MB 2025-02-14 12:03:42,436 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23402.11 MB 2025-02-14 12:03:42,436 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 12:03:42,436 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26696.74 MB 2025-02-14 12:03:42,436 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27640.46 MB 2025-02-14 12:03:42,436 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 12:03:42,436 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24819.54 MB 2025-02-14 12:03:42,647 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 12:03:42,647 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 12:03:42,647 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 12:03:42,647 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:03:42,647 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23402.11 MB 2025-02-14 12:03:42,647 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25643.97 MB 2025-02-14 12:03:42,647 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 12:03:42,647 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27640.46 MB 2025-02-14 12:03:42,647 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33774.63 MB 2025-02-14 12:03:42,647 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 12:03:42,647 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31188.25 MB 2025-02-14 12:03:42,647 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 12:03:42,647 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 12:03:42,647 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 12:03:42,647 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:03:42,647 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21512.58 MB 2025-02-14 12:03:42,647 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25643.97 MB 2025-02-14 12:03:42,647 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 12:03:42,647 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26696.74 MB 2025-02-14 12:03:42,647 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33774.63 MB 2025-02-14 12:03:42,648 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7077.89 MB 2025-02-14 12:03:42,648 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31188.25 MB 2025-02-14 12:03:42,815 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 12:03:42,816 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 12:03:42,816 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 12:03:42,816 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:03:42,816 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27177.51 MB 2025-02-14 12:03:42,816 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27944.51 MB 2025-02-14 12:03:42,816 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 12:03:42,816 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33774.63 MB 2025-02-14 12:03:42,816 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34189.87 MB 2025-02-14 12:03:42,816 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 12:03:42,816 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28652.30 MB 2025-02-14 12:03:42,835 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 12:03:42,835 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 12:03:42,835 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:03:42,835 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:03:42,835 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28357.40 MB 2025-02-14 12:03:42,835 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28585.08 MB 2025-02-14 12:03:42,835 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.67 MB 2025-02-14 12:03:42,835 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34189.87 MB 2025-02-14 12:03:42,835 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34189.87 MB 2025-02-14 12:03:42,835 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:03:42,835 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28796.60 MB 2025-02-14 12:03:42,836 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 12:03:42,836 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 12:03:42,836 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.13 seconds 2025-02-14 12:03:42,836 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:03:42,836 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16456.27 MB 2025-02-14 12:03:42,836 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28785.29 MB 2025-02-14 12:03:42,836 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12329.02 MB 2025-02-14 12:03:42,836 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48102.38 MB 2025-02-14 12:03:42,836 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34189.87 MB 2025-02-14 12:03:42,836 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13912.51 MB 2025-02-14 12:03:42,836 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28796.60 MB 2025-02-14 12:03:43,106 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 12:03:43,106 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 12:03:43,106 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 12:03:43,106 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:03:43,106 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28785.29 MB 2025-02-14 12:03:43,106 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21447.33 MB 2025-02-14 12:03:43,106 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7337.96 MB 2025-02-14 12:03:43,106 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34189.87 MB 2025-02-14 12:03:43,106 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34189.87 MB 2025-02-14 12:03:43,106 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:03:43,106 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31286.20 MB 2025-02-14 12:03:43,124 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8127, cut from 8129 2025-02-14 12:03:43,125 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 12:03:43,131 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 12:03:43,131 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 12:03:43,131 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:03:43,131 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:03:43,131 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21447.33 MB 2025-02-14 12:03:43,131 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29850.89 MB 2025-02-14 12:03:43,131 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8403.56 MB 2025-02-14 12:03:43,131 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34189.87 MB 2025-02-14 12:03:43,131 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42544.92 MB 2025-02-14 12:03:43,131 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8355.05 MB 2025-02-14 12:03:43,131 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29850.89 MB 2025-02-14 12:03:43,294 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7919] 2025-02-14 12:03:43,295 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:03:43,295 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 12:03:43,296 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:03:43,296 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 12:03:43,301 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 12:03:43,302 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:03:43,302 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 12:03:43,302 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 12:03:58,892 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:03:58,892 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 12:03:58,897 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 12:03:58,901 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:03:58,901 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 212, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 12:03:58,902 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:03:58,902 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 212, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 12:04:02,241 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 12:04:02,241 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 12:04:02,241 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.34 seconds 2025-02-14 12:04:02,241 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:04:02,241 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14445.96 MB 2025-02-14 12:04:02,241 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15196.21 MB 2025-02-14 12:04:02,241 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 750.26 MB 2025-02-14 12:04:02,241 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50899.98 MB 2025-02-14 12:04:02,242 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17853.05 MB 2025-02-14 12:04:02,242 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33046.92 MB 2025-02-14 12:04:02,242 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24144.63 MB 2025-02-14 12:04:02,257 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 12:04:02,257 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 12:04:02,257 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:04:02,257 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:04:02,257 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15196.21 MB 2025-02-14 12:04:02,257 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15440.32 MB 2025-02-14 12:04:02,257 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 244.11 MB 2025-02-14 12:04:02,257 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17853.05 MB 2025-02-14 12:04:02,257 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19237.18 MB 2025-02-14 12:04:02,257 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1384.12 MB 2025-02-14 12:04:02,257 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17950.07 MB 2025-02-14 12:04:03,215 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 12:04:03,215 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 12:04:03,215 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.96 seconds 2025-02-14 12:04:03,215 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:04:03,215 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15440.32 MB 2025-02-14 12:04:03,215 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15699.10 MB 2025-02-14 12:04:03,215 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 258.79 MB 2025-02-14 12:04:03,215 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19237.18 MB 2025-02-14 12:04:03,215 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18475.91 MB 2025-02-14 12:04:03,215 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -761.27 MB 2025-02-14 12:04:03,215 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19696.73 MB 2025-02-14 12:04:03,224 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 12:04:03,224 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 12:04:03,224 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:04:03,224 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:04:03,224 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15699.04 MB 2025-02-14 12:04:03,224 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16619.96 MB 2025-02-14 12:04:03,224 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 920.92 MB 2025-02-14 12:04:03,224 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18475.91 MB 2025-02-14 12:04:03,224 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18937.28 MB 2025-02-14 12:04:03,224 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 461.37 MB 2025-02-14 12:04:03,224 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17310.96 MB 2025-02-14 12:04:03,331 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 12:04:03,331 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 12:04:03,331 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 12:04:03,331 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:04:03,331 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16619.96 MB 2025-02-14 12:04:03,331 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17712.90 MB 2025-02-14 12:04:03,331 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1092.94 MB 2025-02-14 12:04:03,331 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18937.28 MB 2025-02-14 12:04:03,331 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21705.52 MB 2025-02-14 12:04:03,331 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2768.24 MB 2025-02-14 12:04:03,331 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20415.70 MB 2025-02-14 12:04:03,332 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 12:04:03,332 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 12:04:03,332 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 12:04:03,332 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:04:03,332 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15699.04 MB 2025-02-14 12:04:03,332 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17712.90 MB 2025-02-14 12:04:03,332 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2013.86 MB 2025-02-14 12:04:03,332 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18475.91 MB 2025-02-14 12:04:03,332 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21705.52 MB 2025-02-14 12:04:03,332 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3229.61 MB 2025-02-14 12:04:03,332 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20415.70 MB 2025-02-14 12:04:03,413 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 12:04:03,413 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 12:04:03,413 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 12:04:03,413 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:04:03,413 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18460.50 MB 2025-02-14 12:04:03,413 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18834.42 MB 2025-02-14 12:04:03,413 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 373.91 MB 2025-02-14 12:04:03,413 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21705.52 MB 2025-02-14 12:04:03,413 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21904.75 MB 2025-02-14 12:04:03,413 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 199.23 MB 2025-02-14 12:04:03,413 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19184.18 MB 2025-02-14 12:04:03,424 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 12:04:03,424 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 12:04:03,424 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:04:03,424 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:04:03,424 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19035.71 MB 2025-02-14 12:04:03,424 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19244.41 MB 2025-02-14 12:04:03,424 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 208.71 MB 2025-02-14 12:04:03,424 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21904.75 MB 2025-02-14 12:04:03,424 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21904.75 MB 2025-02-14 12:04:03,424 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:04:03,424 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19289.35 MB 2025-02-14 12:04:03,425 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 12:04:03,425 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 12:04:03,425 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.52 seconds 2025-02-14 12:04:03,425 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:04:03,425 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13707.33 MB 2025-02-14 12:04:03,425 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19445.00 MB 2025-02-14 12:04:03,425 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5737.66 MB 2025-02-14 12:04:03,425 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50899.98 MB 2025-02-14 12:04:03,425 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21904.75 MB 2025-02-14 12:04:03,426 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28995.22 MB 2025-02-14 12:04:03,426 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19445.00 MB 2025-02-14 12:04:03,694 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 12:04:03,694 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 12:04:03,694 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 12:04:03,694 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:04:03,694 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14729.85 MB 2025-02-14 12:04:03,694 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17736.52 MB 2025-02-14 12:04:03,694 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3006.66 MB 2025-02-14 12:04:03,694 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21904.75 MB 2025-02-14 12:04:03,694 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21904.75 MB 2025-02-14 12:04:03,694 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:04:03,694 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18037.15 MB 2025-02-14 12:04:03,712 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8142, cut from 8144 2025-02-14 12:04:03,712 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2.'] 2025-02-14 12:04:03,718 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 12:04:03,718 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 12:04:03,718 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:04:03,718 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:04:03,718 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17736.52 MB 2025-02-14 12:04:03,718 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26154.67 MB 2025-02-14 12:04:03,718 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8418.15 MB 2025-02-14 12:04:03,718 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21904.75 MB 2025-02-14 12:04:03,718 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32367.44 MB 2025-02-14 12:04:03,718 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10462.69 MB 2025-02-14 12:04:03,718 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26154.67 MB 2025-02-14 12:04:03,884 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7934] 2025-02-14 12:04:03,885 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:04:03,886 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 12:04:03,886 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:04:03,886 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 12:04:03,891 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 12:04:03,893 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:04:03,893 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 12:04:03,893 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2.'] 2025-02-14 12:05:08,607 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:05:08,607 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 12:05:08,613 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 12:05:08,618 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:05:08,618 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 257, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 12:05:08,620 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:05:08,620 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 257, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 12:05:12,621 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 12:05:12,621 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 12:05:12,621 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.00 seconds 2025-02-14 12:05:12,621 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:05:12,621 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14759.52 MB 2025-02-14 12:05:12,621 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15669.03 MB 2025-02-14 12:05:12,621 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 909.51 MB 2025-02-14 12:05:12,621 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44921.00 MB 2025-02-14 12:05:12,621 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18293.46 MB 2025-02-14 12:05:12,621 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26627.54 MB 2025-02-14 12:05:12,621 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24684.69 MB 2025-02-14 12:05:12,640 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 12:05:12,640 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 12:05:12,640 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:05:12,640 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:05:12,640 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15669.03 MB 2025-02-14 12:05:12,640 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16110.28 MB 2025-02-14 12:05:12,640 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 441.24 MB 2025-02-14 12:05:12,640 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18293.46 MB 2025-02-14 12:05:12,640 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20998.78 MB 2025-02-14 12:05:12,640 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2705.33 MB 2025-02-14 12:05:12,640 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19283.72 MB 2025-02-14 12:05:13,871 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 12:05:13,871 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 12:05:13,871 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.23 seconds 2025-02-14 12:05:13,872 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:05:13,872 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16110.28 MB 2025-02-14 12:05:13,872 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16451.34 MB 2025-02-14 12:05:13,872 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 341.07 MB 2025-02-14 12:05:13,872 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20998.78 MB 2025-02-14 12:05:13,872 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19184.75 MB 2025-02-14 12:05:13,872 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1814.04 MB 2025-02-14 12:05:13,872 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20451.62 MB 2025-02-14 12:05:13,881 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 12:05:13,881 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 12:05:13,881 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:05:13,881 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:05:13,881 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16451.34 MB 2025-02-14 12:05:13,881 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17665.16 MB 2025-02-14 12:05:13,881 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1213.82 MB 2025-02-14 12:05:13,881 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19184.75 MB 2025-02-14 12:05:13,881 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19792.92 MB 2025-02-14 12:05:13,881 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 608.17 MB 2025-02-14 12:05:13,881 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18575.86 MB 2025-02-14 12:05:14,018 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 12:05:14,018 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 12:05:14,018 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-14 12:05:14,018 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:05:14,018 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17665.16 MB 2025-02-14 12:05:14,018 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19105.57 MB 2025-02-14 12:05:14,018 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1440.42 MB 2025-02-14 12:05:14,018 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19792.92 MB 2025-02-14 12:05:14,018 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23899.14 MB 2025-02-14 12:05:14,018 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4106.22 MB 2025-02-14 12:05:14,019 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22669.59 MB 2025-02-14 12:05:14,019 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 12:05:14,019 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 12:05:14,019 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 12:05:14,019 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:05:14,019 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16451.34 MB 2025-02-14 12:05:14,020 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19105.57 MB 2025-02-14 12:05:14,020 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2654.23 MB 2025-02-14 12:05:14,020 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19184.75 MB 2025-02-14 12:05:14,020 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23899.14 MB 2025-02-14 12:05:14,020 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4714.40 MB 2025-02-14 12:05:14,020 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22669.59 MB 2025-02-14 12:05:14,132 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 12:05:14,132 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 12:05:14,132 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 12:05:14,132 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:05:14,132 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20090.88 MB 2025-02-14 12:05:14,132 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20585.51 MB 2025-02-14 12:05:14,132 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 494.63 MB 2025-02-14 12:05:14,132 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23899.14 MB 2025-02-14 12:05:14,132 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24163.39 MB 2025-02-14 12:05:14,132 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 264.24 MB 2025-02-14 12:05:14,132 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21040.26 MB 2025-02-14 12:05:14,145 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 12:05:14,145 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 12:05:14,145 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:05:14,145 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:05:14,145 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20850.80 MB 2025-02-14 12:05:14,145 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21081.37 MB 2025-02-14 12:05:14,145 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 230.58 MB 2025-02-14 12:05:14,145 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24163.39 MB 2025-02-14 12:05:14,145 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24163.39 MB 2025-02-14 12:05:14,145 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:05:14,145 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21176.25 MB 2025-02-14 12:05:14,146 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 12:05:14,147 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 12:05:14,147 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.52 seconds 2025-02-14 12:05:14,147 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:05:14,147 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13864.11 MB 2025-02-14 12:05:14,147 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21282.45 MB 2025-02-14 12:05:14,147 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7418.33 MB 2025-02-14 12:05:14,147 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44921.00 MB 2025-02-14 12:05:14,147 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24163.39 MB 2025-02-14 12:05:14,147 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20757.61 MB 2025-02-14 12:05:14,147 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21282.45 MB 2025-02-14 12:05:14,414 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 12:05:14,414 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 12:05:14,414 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 12:05:14,414 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:05:14,414 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21282.45 MB 2025-02-14 12:05:14,414 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24296.48 MB 2025-02-14 12:05:14,414 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-14 12:05:14,414 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24163.39 MB 2025-02-14 12:05:14,414 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25908.22 MB 2025-02-14 12:05:14,414 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1744.83 MB 2025-02-14 12:05:14,414 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24598.11 MB 2025-02-14 12:05:14,432 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 12:05:14,433 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 12:05:14,439 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 12:05:14,439 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 12:05:14,439 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:05:14,439 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:05:14,439 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18194.89 MB 2025-02-14 12:05:14,439 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26633.91 MB 2025-02-14 12:05:14,439 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 12:05:14,439 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25908.22 MB 2025-02-14 12:05:14,439 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36398.17 MB 2025-02-14 12:05:14,439 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 12:05:14,439 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26633.91 MB 2025-02-14 12:05:14,602 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 12:05:14,603 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:05:14,603 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 12:05:14,604 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:05:14,604 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 12:05:14,609 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 12:05:14,610 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:05:14,610 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 12:05:14,610 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 12:05:22,960 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:05:22,960 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 12:05:22,965 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 12:05:22,968 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:05:22,968 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1444, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 12:05:22,969 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:05:22,969 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1444, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 12:05:45,177 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 12:05:45,177 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 12:05:45,177 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.20 seconds 2025-02-14 12:05:45,177 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:05:45,177 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23030.73 MB 2025-02-14 12:05:45,177 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28141.49 MB 2025-02-14 12:05:45,177 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5110.76 MB 2025-02-14 12:05:45,177 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48983.18 MB 2025-02-14 12:05:45,177 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38801.51 MB 2025-02-14 12:05:45,177 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10181.67 MB 2025-02-14 12:05:45,177 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37031.95 MB 2025-02-14 12:05:45,256 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 12:05:45,256 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 12:05:45,256 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 12:05:45,256 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:05:45,256 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28141.49 MB 2025-02-14 12:05:45,256 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23284.76 MB 2025-02-14 12:05:45,256 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4856.73 MB 2025-02-14 12:05:45,256 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38801.51 MB 2025-02-14 12:05:45,256 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48473.57 MB 2025-02-14 12:05:45,256 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9672.07 MB 2025-02-14 12:05:45,256 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42702.91 MB 2025-02-14 12:05:47,171 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 12:05:47,172 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 12:05:47,172 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 12:05:47,172 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:05:47,172 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23284.76 MB 2025-02-14 12:05:47,172 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23815.60 MB 2025-02-14 12:05:47,172 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 12:05:47,172 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48473.57 MB 2025-02-14 12:05:47,172 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33690.75 MB 2025-02-14 12:05:47,172 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14782.82 MB 2025-02-14 12:05:47,172 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27794.93 MB 2025-02-14 12:05:47,185 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 12:05:47,185 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 12:05:47,185 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:05:47,185 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:05:47,185 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23815.60 MB 2025-02-14 12:05:47,185 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25705.13 MB 2025-02-14 12:05:47,185 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 12:05:47,185 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33690.75 MB 2025-02-14 12:05:47,185 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33690.75 MB 2025-02-14 12:05:47,185 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:05:47,185 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27122.56 MB 2025-02-14 12:05:47,395 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 12:05:47,395 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 12:05:47,395 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 12:05:47,395 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:05:47,395 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25705.13 MB 2025-02-14 12:05:47,395 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27946.99 MB 2025-02-14 12:05:47,395 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 12:05:47,395 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33690.75 MB 2025-02-14 12:05:47,395 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37465.62 MB 2025-02-14 12:05:47,395 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 12:05:47,395 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33491.27 MB 2025-02-14 12:05:47,396 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 12:05:47,396 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 12:05:47,396 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 12:05:47,396 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:05:47,396 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23815.60 MB 2025-02-14 12:05:47,396 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27946.99 MB 2025-02-14 12:05:47,396 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 12:05:47,396 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33690.75 MB 2025-02-14 12:05:47,396 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37465.62 MB 2025-02-14 12:05:47,396 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 12:05:47,396 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33491.27 MB 2025-02-14 12:05:47,565 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 12:05:47,565 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 12:05:47,565 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 12:05:47,565 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:05:47,565 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29480.53 MB 2025-02-14 12:05:47,565 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30247.53 MB 2025-02-14 12:05:47,565 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 12:05:47,565 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37465.62 MB 2025-02-14 12:05:47,565 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37878.76 MB 2025-02-14 12:05:47,565 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 12:05:47,565 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30955.32 MB 2025-02-14 12:05:47,584 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 12:05:47,584 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 12:05:47,584 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:05:47,584 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:05:47,584 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30660.42 MB 2025-02-14 12:05:47,584 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30887.54 MB 2025-02-14 12:05:47,584 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.12 MB 2025-02-14 12:05:47,584 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37878.76 MB 2025-02-14 12:05:47,584 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37878.76 MB 2025-02-14 12:05:47,584 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:05:47,584 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31111.56 MB 2025-02-14 12:05:47,585 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 12:05:47,585 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 12:05:47,585 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.61 seconds 2025-02-14 12:05:47,585 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:05:47,585 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17999.72 MB 2025-02-14 12:05:47,585 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31087.58 MB 2025-02-14 12:05:47,585 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13087.86 MB 2025-02-14 12:05:47,585 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48983.18 MB 2025-02-14 12:05:47,585 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37878.76 MB 2025-02-14 12:05:47,585 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11104.42 MB 2025-02-14 12:05:47,585 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31111.56 MB 2025-02-14 12:05:47,856 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 12:05:47,856 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 12:05:47,856 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 12:05:47,856 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:05:47,856 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31087.58 MB 2025-02-14 12:05:47,856 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22988.11 MB 2025-02-14 12:05:47,856 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8099.47 MB 2025-02-14 12:05:47,856 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37878.76 MB 2025-02-14 12:05:47,856 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37878.76 MB 2025-02-14 12:05:47,856 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:05:47,856 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33586.34 MB 2025-02-14 12:05:47,911 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8120, cut from 8122 2025-02-14 12:05:47,911 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 12:05:47,921 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 12:05:47,921 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 12:05:47,921 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 12:05:47,921 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:05:47,921 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22988.11 MB 2025-02-14 12:05:47,921 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31384.75 MB 2025-02-14 12:05:47,921 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8396.64 MB 2025-02-14 12:05:47,921 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37878.76 MB 2025-02-14 12:05:47,921 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46225.42 MB 2025-02-14 12:05:47,921 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8346.66 MB 2025-02-14 12:05:47,921 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31384.75 MB 2025-02-14 12:05:48,082 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7912] 2025-02-14 12:05:48,083 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:05:48,083 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 12:05:48,084 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:05:48,084 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 12:05:48,089 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 12:05:48,090 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:05:48,090 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 12:05:48,090 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 12:06:38,979 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:06:38,979 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 12:06:38,984 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 12:06:38,988 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:06:38,988 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 151, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 12:06:38,989 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:06:38,989 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 151, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 12:06:41,341 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 12:06:41,341 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 12:06:41,341 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.35 seconds 2025-02-14 12:06:41,341 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:06:41,341 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14020.90 MB 2025-02-14 12:06:41,341 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14555.28 MB 2025-02-14 12:06:41,341 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 534.38 MB 2025-02-14 12:06:41,341 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54572.09 MB 2025-02-14 12:06:41,341 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 16911.43 MB 2025-02-14 12:06:41,341 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -37660.66 MB 2025-02-14 12:06:41,341 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23493.08 MB 2025-02-14 12:06:41,353 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 12:06:41,353 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 12:06:41,353 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:06:41,353 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:06:41,353 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14555.28 MB 2025-02-14 12:06:41,353 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14772.70 MB 2025-02-14 12:06:41,353 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 217.42 MB 2025-02-14 12:06:41,353 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 16911.43 MB 2025-02-14 12:06:41,353 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17683.19 MB 2025-02-14 12:06:41,353 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 771.75 MB 2025-02-14 12:06:41,353 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16617.45 MB 2025-02-14 12:06:42,056 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 12:06:42,056 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 12:06:42,056 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.70 seconds 2025-02-14 12:06:42,056 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:06:42,056 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14772.70 MB 2025-02-14 12:06:42,056 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14965.13 MB 2025-02-14 12:06:42,056 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 192.43 MB 2025-02-14 12:06:42,056 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17683.19 MB 2025-02-14 12:06:42,056 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17683.19 MB 2025-02-14 12:06:42,056 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:06:42,056 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18944.18 MB 2025-02-14 12:06:42,063 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 12:06:42,063 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 12:06:42,063 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 12:06:42,063 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:06:42,063 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14965.07 MB 2025-02-14 12:06:42,063 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15649.86 MB 2025-02-14 12:06:42,063 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 684.79 MB 2025-02-14 12:06:42,063 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17683.19 MB 2025-02-14 12:06:42,064 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17683.19 MB 2025-02-14 12:06:42,064 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:06:42,064 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16163.68 MB 2025-02-14 12:06:42,143 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 12:06:42,143 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 12:06:42,143 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 12:06:42,143 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:06:42,143 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15649.86 MB 2025-02-14 12:06:42,143 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16462.57 MB 2025-02-14 12:06:42,143 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 812.71 MB 2025-02-14 12:06:42,143 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17683.19 MB 2025-02-14 12:06:42,143 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19402.85 MB 2025-02-14 12:06:42,143 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1719.66 MB 2025-02-14 12:06:42,143 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18472.33 MB 2025-02-14 12:06:42,144 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 12:06:42,144 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 12:06:42,144 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 12:06:42,144 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:06:42,144 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14965.07 MB 2025-02-14 12:06:42,144 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16462.57 MB 2025-02-14 12:06:42,144 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1497.50 MB 2025-02-14 12:06:42,144 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17683.19 MB 2025-02-14 12:06:42,144 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19402.85 MB 2025-02-14 12:06:42,144 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1719.66 MB 2025-02-14 12:06:42,144 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18472.33 MB 2025-02-14 12:06:42,206 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 12:06:42,206 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 12:06:42,206 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 12:06:42,206 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:06:42,206 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17018.48 MB 2025-02-14 12:06:42,206 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17296.52 MB 2025-02-14 12:06:42,206 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 278.04 MB 2025-02-14 12:06:42,206 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19402.85 MB 2025-02-14 12:06:42,206 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19547.55 MB 2025-02-14 12:06:42,206 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 144.70 MB 2025-02-14 12:06:42,206 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17563.34 MB 2025-02-14 12:06:42,214 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 12:06:42,214 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 12:06:42,214 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:06:42,214 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:06:42,215 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17446.20 MB 2025-02-14 12:06:42,215 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17672.71 MB 2025-02-14 12:06:42,215 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.51 MB 2025-02-14 12:06:42,215 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19547.55 MB 2025-02-14 12:06:42,215 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19547.55 MB 2025-02-14 12:06:42,215 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:06:42,215 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17674.57 MB 2025-02-14 12:06:42,216 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 12:06:42,216 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 12:06:42,216 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.22 seconds 2025-02-14 12:06:42,216 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:06:42,216 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13494.80 MB 2025-02-14 12:06:42,216 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17873.39 MB 2025-02-14 12:06:42,216 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4378.59 MB 2025-02-14 12:06:42,216 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54572.09 MB 2025-02-14 12:06:42,216 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19547.55 MB 2025-02-14 12:06:42,216 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35024.54 MB 2025-02-14 12:06:42,216 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17873.39 MB 2025-02-14 12:06:42,482 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 12:06:42,482 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 12:06:42,482 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 12:06:42,482 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:06:42,482 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17873.39 MB 2025-02-14 12:06:42,482 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17289.68 MB 2025-02-14 12:06:42,482 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -583.71 MB 2025-02-14 12:06:42,482 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19547.55 MB 2025-02-14 12:06:42,482 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19815.99 MB 2025-02-14 12:06:42,482 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 268.44 MB 2025-02-14 12:06:42,482 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18976.89 MB 2025-02-14 12:06:42,501 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8146, cut from 8148 2025-02-14 12:06:42,501 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2,'] 2025-02-14 12:06:42,507 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 12:06:42,507 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 12:06:42,507 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:06:42,507 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:06:42,507 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17289.68 MB 2025-02-14 12:06:42,507 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25712.00 MB 2025-02-14 12:06:42,507 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8422.33 MB 2025-02-14 12:06:42,507 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19815.99 MB 2025-02-14 12:06:42,507 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30284.97 MB 2025-02-14 12:06:42,507 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10468.98 MB 2025-02-14 12:06:42,507 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25712.00 MB 2025-02-14 12:06:42,670 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7938] 2025-02-14 12:06:42,671 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:06:42,671 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 12:06:42,672 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:06:42,672 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 12:06:42,677 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 12:06:42,678 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:06:42,678 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 12:06:42,678 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2,'] 2025-02-14 12:07:05,894 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:07:05,894 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 12:07:05,899 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 12:07:05,903 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:07:05,903 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1136, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 12:07:05,904 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:07:05,904 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1136, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 12:07:23,379 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 12:07:23,379 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 12:07:23,379 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.47 seconds 2025-02-14 12:07:23,379 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:07:23,379 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20884.54 MB 2025-02-14 12:07:23,379 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24904.78 MB 2025-02-14 12:07:23,379 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4020.24 MB 2025-02-14 12:07:23,379 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42844.82 MB 2025-02-14 12:07:23,379 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31398.56 MB 2025-02-14 12:07:23,379 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11446.26 MB 2025-02-14 12:07:23,379 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33753.29 MB 2025-02-14 12:07:23,452 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 12:07:23,452 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 12:07:23,452 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 12:07:23,452 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:07:23,452 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24904.78 MB 2025-02-14 12:07:23,452 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21683.56 MB 2025-02-14 12:07:23,452 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3221.22 MB 2025-02-14 12:07:23,452 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31398.56 MB 2025-02-14 12:07:23,452 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42580.57 MB 2025-02-14 12:07:23,452 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 11182.01 MB 2025-02-14 12:07:23,452 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37166.64 MB 2025-02-14 12:07:25,370 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 12:07:25,370 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 12:07:25,370 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 12:07:25,370 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:07:25,370 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21683.56 MB 2025-02-14 12:07:25,370 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22214.40 MB 2025-02-14 12:07:25,370 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 12:07:25,370 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42580.57 MB 2025-02-14 12:07:25,370 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28793.90 MB 2025-02-14 12:07:25,370 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13786.68 MB 2025-02-14 12:07:25,370 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26193.74 MB 2025-02-14 12:07:25,384 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 12:07:25,384 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 12:07:25,384 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:07:25,384 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:07:25,384 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22214.40 MB 2025-02-14 12:07:25,384 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24103.94 MB 2025-02-14 12:07:25,384 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 12:07:25,384 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28793.90 MB 2025-02-14 12:07:25,384 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28793.90 MB 2025-02-14 12:07:25,384 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:07:25,384 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25521.36 MB 2025-02-14 12:07:25,596 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 12:07:25,596 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 12:07:25,596 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 12:07:25,596 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:07:25,596 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24103.94 MB 2025-02-14 12:07:25,596 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26345.79 MB 2025-02-14 12:07:25,596 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 12:07:25,596 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28793.90 MB 2025-02-14 12:07:25,596 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34456.21 MB 2025-02-14 12:07:25,596 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 12:07:25,596 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31890.07 MB 2025-02-14 12:07:25,597 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 12:07:25,597 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 12:07:25,597 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 12:07:25,597 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:07:25,597 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22214.40 MB 2025-02-14 12:07:25,597 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26345.79 MB 2025-02-14 12:07:25,597 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 12:07:25,597 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28793.90 MB 2025-02-14 12:07:25,597 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34456.21 MB 2025-02-14 12:07:25,597 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 12:07:25,597 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31890.07 MB 2025-02-14 12:07:25,795 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 12:07:25,795 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 12:07:25,795 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.19 seconds 2025-02-14 12:07:25,795 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:07:25,795 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27879.33 MB 2025-02-14 12:07:25,795 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28646.34 MB 2025-02-14 12:07:25,795 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 12:07:25,795 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34456.21 MB 2025-02-14 12:07:25,795 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34869.35 MB 2025-02-14 12:07:25,795 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 12:07:25,795 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29354.12 MB 2025-02-14 12:07:25,814 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 12:07:25,814 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 12:07:25,814 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:07:25,814 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:07:25,814 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29059.23 MB 2025-02-14 12:07:25,814 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29288.08 MB 2025-02-14 12:07:25,814 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.86 MB 2025-02-14 12:07:25,814 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34869.35 MB 2025-02-14 12:07:25,814 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34869.35 MB 2025-02-14 12:07:25,814 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:07:25,814 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29516.62 MB 2025-02-14 12:07:25,815 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 12:07:25,815 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 12:07:25,815 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.91 seconds 2025-02-14 12:07:25,815 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:07:25,815 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16926.62 MB 2025-02-14 12:07:25,815 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29488.39 MB 2025-02-14 12:07:25,815 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12561.77 MB 2025-02-14 12:07:25,815 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42844.82 MB 2025-02-14 12:07:25,815 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34869.35 MB 2025-02-14 12:07:25,816 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7975.47 MB 2025-02-14 12:07:25,816 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29516.62 MB 2025-02-14 12:07:26,083 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 12:07:26,083 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 12:07:26,083 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 12:07:26,083 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:07:26,083 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29488.39 MB 2025-02-14 12:07:26,083 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21919.20 MB 2025-02-14 12:07:26,083 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7569.19 MB 2025-02-14 12:07:26,083 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34869.35 MB 2025-02-14 12:07:26,083 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34869.35 MB 2025-02-14 12:07:26,083 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:07:26,083 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31990.54 MB 2025-02-14 12:07:26,101 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8131, cut from 8133 2025-02-14 12:07:26,102 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 12:07:26,108 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 12:07:26,108 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 12:07:26,108 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:07:26,108 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:07:26,108 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21919.20 MB 2025-02-14 12:07:26,108 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30327.32 MB 2025-02-14 12:07:26,108 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8408.12 MB 2025-02-14 12:07:26,108 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34869.35 MB 2025-02-14 12:07:26,108 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43228.59 MB 2025-02-14 12:07:26,108 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8359.25 MB 2025-02-14 12:07:26,108 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30327.32 MB 2025-02-14 12:07:26,269 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7923] 2025-02-14 12:07:26,270 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:07:26,271 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 12:07:26,271 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:07:26,272 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 12:07:26,276 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 12:07:26,277 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:07:26,277 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 12:07:26,277 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 12:08:20,036 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:08:20,036 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 12:08:20,041 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 12:08:20,045 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:08:20,045 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 420, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 12:08:20,046 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:08:20,046 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 420, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 12:08:26,540 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 12:08:26,540 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 12:08:26,540 - resource_logging.py:150 - __exit__ - DEBUG - Time: 6.49 seconds 2025-02-14 12:08:26,540 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:08:26,540 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15895.33 MB 2025-02-14 12:08:26,540 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17382.21 MB 2025-02-14 12:08:26,540 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1486.88 MB 2025-02-14 12:08:26,540 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51587.84 MB 2025-02-14 12:08:26,540 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20285.75 MB 2025-02-14 12:08:26,540 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31302.09 MB 2025-02-14 12:08:26,540 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26273.48 MB 2025-02-14 12:08:26,572 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 12:08:26,572 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 12:08:26,572 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 12:08:26,572 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:08:26,572 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17382.21 MB 2025-02-14 12:08:26,572 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17962.35 MB 2025-02-14 12:08:26,572 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 580.13 MB 2025-02-14 12:08:26,572 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20285.75 MB 2025-02-14 12:08:26,572 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27562.87 MB 2025-02-14 12:08:26,572 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7277.12 MB 2025-02-14 12:08:26,572 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24467.77 MB 2025-02-14 12:08:28,484 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 12:08:28,484 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 12:08:28,484 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 12:08:28,484 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:08:28,484 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17962.35 MB 2025-02-14 12:08:28,484 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18493.19 MB 2025-02-14 12:08:28,484 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 12:08:28,484 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27562.87 MB 2025-02-14 12:08:28,484 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19979.57 MB 2025-02-14 12:08:28,484 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7583.30 MB 2025-02-14 12:08:28,484 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22473.56 MB 2025-02-14 12:08:28,498 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 12:08:28,498 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 12:08:28,498 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:08:28,498 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:08:28,498 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18493.19 MB 2025-02-14 12:08:28,498 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20382.72 MB 2025-02-14 12:08:28,498 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 12:08:28,498 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19979.57 MB 2025-02-14 12:08:28,498 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23754.44 MB 2025-02-14 12:08:28,498 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 12:08:28,498 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21800.15 MB 2025-02-14 12:08:28,709 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 12:08:28,709 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 12:08:28,709 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 12:08:28,709 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:08:28,709 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20382.72 MB 2025-02-14 12:08:28,709 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22624.58 MB 2025-02-14 12:08:28,709 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 12:08:28,709 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23754.44 MB 2025-02-14 12:08:28,709 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30360.47 MB 2025-02-14 12:08:28,709 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 12:08:28,709 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28168.86 MB 2025-02-14 12:08:28,710 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 12:08:28,710 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 12:08:28,710 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 12:08:28,710 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:08:28,710 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18493.19 MB 2025-02-14 12:08:28,710 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22624.58 MB 2025-02-14 12:08:28,710 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 12:08:28,710 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19979.57 MB 2025-02-14 12:08:28,710 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30360.47 MB 2025-02-14 12:08:28,710 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10380.90 MB 2025-02-14 12:08:28,710 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28168.86 MB 2025-02-14 12:08:28,880 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 12:08:28,880 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 12:08:28,880 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 12:08:28,880 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:08:28,880 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24158.12 MB 2025-02-14 12:08:28,880 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24925.12 MB 2025-02-14 12:08:28,880 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 12:08:28,880 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30360.47 MB 2025-02-14 12:08:28,880 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30773.61 MB 2025-02-14 12:08:28,880 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 12:08:28,880 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25632.91 MB 2025-02-14 12:08:28,898 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 12:08:28,898 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 12:08:28,898 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:08:28,898 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:08:28,898 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25338.01 MB 2025-02-14 12:08:28,898 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25566.19 MB 2025-02-14 12:08:28,898 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.17 MB 2025-02-14 12:08:28,898 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30773.61 MB 2025-02-14 12:08:28,899 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30773.61 MB 2025-02-14 12:08:28,899 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:08:28,899 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25789.89 MB 2025-02-14 12:08:28,900 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 12:08:28,900 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 12:08:28,900 - resource_logging.py:150 - __exit__ - DEBUG - Time: 8.85 seconds 2025-02-14 12:08:28,900 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:08:28,900 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14432.02 MB 2025-02-14 12:08:28,900 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25766.28 MB 2025-02-14 12:08:28,900 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11334.26 MB 2025-02-14 12:08:28,900 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51587.84 MB 2025-02-14 12:08:28,900 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30773.61 MB 2025-02-14 12:08:28,900 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20814.23 MB 2025-02-14 12:08:28,900 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25789.89 MB 2025-02-14 12:08:29,166 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 12:08:29,166 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 12:08:29,166 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 12:08:29,166 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:08:29,166 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25766.28 MB 2025-02-14 12:08:29,166 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19421.17 MB 2025-02-14 12:08:29,166 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6345.10 MB 2025-02-14 12:08:29,166 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30773.61 MB 2025-02-14 12:08:29,166 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30773.61 MB 2025-02-14 12:08:29,166 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:08:29,166 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28265.66 MB 2025-02-14 12:08:29,184 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8122, cut from 8124 2025-02-14 12:08:29,184 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 12:08:29,190 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 12:08:29,190 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 12:08:29,191 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:08:29,191 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:08:29,191 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19421.17 MB 2025-02-14 12:08:29,191 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27818.57 MB 2025-02-14 12:08:29,191 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8397.40 MB 2025-02-14 12:08:29,191 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30773.61 MB 2025-02-14 12:08:29,191 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41213.23 MB 2025-02-14 12:08:29,191 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10439.62 MB 2025-02-14 12:08:29,191 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27818.57 MB 2025-02-14 12:08:29,352 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7914] 2025-02-14 12:08:29,353 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:08:29,353 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 12:08:29,354 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:08:29,354 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 12:08:29,359 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 12:08:29,360 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:08:29,360 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 12:08:29,360 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 12:08:39,118 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:08:39,118 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 12:08:39,123 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 12:08:39,127 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:08:39,127 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1246, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 12:08:39,128 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:08:39,128 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1246, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 12:08:58,405 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 12:08:58,405 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 12:08:58,405 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.27 seconds 2025-02-14 12:08:58,405 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:08:58,405 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21651.03 MB 2025-02-14 12:08:58,405 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26061.34 MB 2025-02-14 12:08:58,405 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4410.31 MB 2025-02-14 12:08:58,405 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49564.09 MB 2025-02-14 12:08:58,405 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38021.37 MB 2025-02-14 12:08:58,405 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11542.72 MB 2025-02-14 12:08:58,405 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34972.78 MB 2025-02-14 12:08:58,479 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 12:08:58,479 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 12:08:58,479 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 12:08:58,479 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:08:58,479 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26061.34 MB 2025-02-14 12:08:58,479 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22255.42 MB 2025-02-14 12:08:58,479 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3805.93 MB 2025-02-14 12:08:58,479 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38021.37 MB 2025-02-14 12:08:58,479 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46774.88 MB 2025-02-14 12:08:58,479 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8753.51 MB 2025-02-14 12:08:58,479 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39272.52 MB 2025-02-14 12:09:00,404 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 12:09:00,404 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 12:09:00,404 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 12:09:00,404 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:09:00,404 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22255.42 MB 2025-02-14 12:09:00,404 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22786.26 MB 2025-02-14 12:09:00,404 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 12:09:00,404 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46774.88 MB 2025-02-14 12:09:00,404 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33611.06 MB 2025-02-14 12:09:00,404 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13163.82 MB 2025-02-14 12:09:00,404 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26765.59 MB 2025-02-14 12:09:00,417 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 12:09:00,417 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 12:09:00,417 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:09:00,417 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:09:00,417 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22786.26 MB 2025-02-14 12:09:00,417 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24675.79 MB 2025-02-14 12:09:00,417 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 12:09:00,418 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33611.06 MB 2025-02-14 12:09:00,418 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33611.06 MB 2025-02-14 12:09:00,418 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:09:00,418 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26093.22 MB 2025-02-14 12:09:00,629 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 12:09:00,629 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 12:09:00,629 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 12:09:00,629 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:09:00,629 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24675.79 MB 2025-02-14 12:09:00,629 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26917.65 MB 2025-02-14 12:09:00,629 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 12:09:00,629 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33611.06 MB 2025-02-14 12:09:00,629 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35498.49 MB 2025-02-14 12:09:00,629 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 12:09:00,629 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32461.93 MB 2025-02-14 12:09:00,630 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 12:09:00,630 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 12:09:00,630 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 12:09:00,630 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:09:00,630 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22786.26 MB 2025-02-14 12:09:00,630 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26917.65 MB 2025-02-14 12:09:00,630 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 12:09:00,630 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33611.06 MB 2025-02-14 12:09:00,630 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35498.49 MB 2025-02-14 12:09:00,630 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 12:09:00,630 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32461.93 MB 2025-02-14 12:09:00,798 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 12:09:00,798 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 12:09:00,798 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 12:09:00,798 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:09:00,798 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28451.19 MB 2025-02-14 12:09:00,798 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29218.19 MB 2025-02-14 12:09:00,798 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 12:09:00,798 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35498.49 MB 2025-02-14 12:09:00,798 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35913.73 MB 2025-02-14 12:09:00,798 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 12:09:00,798 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29925.98 MB 2025-02-14 12:09:00,817 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 12:09:00,818 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 12:09:00,818 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:09:00,818 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:09:00,818 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29631.08 MB 2025-02-14 12:09:00,818 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29860.18 MB 2025-02-14 12:09:00,818 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.10 MB 2025-02-14 12:09:00,818 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35913.73 MB 2025-02-14 12:09:00,818 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35913.73 MB 2025-02-14 12:09:00,818 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:09:00,818 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30089.72 MB 2025-02-14 12:09:00,819 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 12:09:00,819 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 12:09:00,819 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.69 seconds 2025-02-14 12:09:00,819 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:09:00,819 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17309.87 MB 2025-02-14 12:09:00,819 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30061.00 MB 2025-02-14 12:09:00,819 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12751.13 MB 2025-02-14 12:09:00,819 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49564.09 MB 2025-02-14 12:09:00,819 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35913.73 MB 2025-02-14 12:09:00,819 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13650.36 MB 2025-02-14 12:09:00,819 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30089.72 MB 2025-02-14 12:09:01,088 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 12:09:01,088 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 12:09:01,088 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 12:09:01,088 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:09:01,088 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30061.00 MB 2025-02-14 12:09:01,088 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22310.45 MB 2025-02-14 12:09:01,088 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7750.55 MB 2025-02-14 12:09:01,088 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35913.73 MB 2025-02-14 12:09:01,088 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35913.73 MB 2025-02-14 12:09:01,088 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:09:01,088 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32569.60 MB 2025-02-14 12:09:01,106 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8152, cut from 8154 2025-02-14 12:09:01,107 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 12:09:01,113 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 12:09:01,113 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 12:09:01,113 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:09:01,113 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:09:01,113 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22310.45 MB 2025-02-14 12:09:01,113 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30739.57 MB 2025-02-14 12:09:01,113 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8429.12 MB 2025-02-14 12:09:01,113 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35913.73 MB 2025-02-14 12:09:01,113 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44293.95 MB 2025-02-14 12:09:01,113 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8380.22 MB 2025-02-14 12:09:01,113 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30739.57 MB 2025-02-14 12:09:01,278 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7944] 2025-02-14 12:09:01,280 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:09:01,280 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 12:09:01,281 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:09:01,281 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 12:09:01,285 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 12:09:01,286 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:09:01,287 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 12:09:01,287 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 12:09:58,015 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:09:58,015 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 12:09:58,020 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 12:09:58,024 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:09:58,024 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 177, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 12:09:58,025 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:09:58,025 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 177, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 12:10:00,761 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 12:10:00,761 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 12:10:00,761 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.73 seconds 2025-02-14 12:10:00,761 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:10:00,761 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14202.07 MB 2025-02-14 12:10:00,761 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14828.46 MB 2025-02-14 12:10:00,761 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 626.39 MB 2025-02-14 12:10:00,761 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52674.17 MB 2025-02-14 12:10:00,761 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17381.20 MB 2025-02-14 12:10:00,761 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35292.97 MB 2025-02-14 12:10:00,761 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23674.25 MB 2025-02-14 12:10:00,775 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 12:10:00,775 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 12:10:00,775 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:10:00,775 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:10:00,775 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14828.46 MB 2025-02-14 12:10:00,775 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15026.60 MB 2025-02-14 12:10:00,775 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 198.14 MB 2025-02-14 12:10:00,775 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17381.20 MB 2025-02-14 12:10:00,775 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18530.44 MB 2025-02-14 12:10:00,775 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1149.24 MB 2025-02-14 12:10:00,775 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17107.53 MB 2025-02-14 12:10:01,558 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 12:10:01,558 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 12:10:01,558 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.78 seconds 2025-02-14 12:10:01,558 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:10:01,558 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15026.60 MB 2025-02-14 12:10:01,558 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15241.60 MB 2025-02-14 12:10:01,558 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 214.99 MB 2025-02-14 12:10:01,558 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18530.44 MB 2025-02-14 12:10:01,558 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17955.82 MB 2025-02-14 12:10:01,558 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -574.62 MB 2025-02-14 12:10:01,558 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19198.08 MB 2025-02-14 12:10:01,567 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 12:10:01,567 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 12:10:01,567 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 12:10:01,567 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:10:01,567 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15241.53 MB 2025-02-14 12:10:01,567 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16006.61 MB 2025-02-14 12:10:01,567 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 765.08 MB 2025-02-14 12:10:01,567 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17955.82 MB 2025-02-14 12:10:01,567 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17955.82 MB 2025-02-14 12:10:01,567 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:10:01,567 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16580.67 MB 2025-02-14 12:10:01,657 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 12:10:01,657 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 12:10:01,657 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 12:10:01,657 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:10:01,657 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16006.61 MB 2025-02-14 12:10:01,657 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16914.60 MB 2025-02-14 12:10:01,657 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 907.99 MB 2025-02-14 12:10:01,657 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17955.82 MB 2025-02-14 12:10:01,657 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20260.59 MB 2025-02-14 12:10:01,657 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2304.77 MB 2025-02-14 12:10:01,657 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19159.99 MB 2025-02-14 12:10:01,657 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 12:10:01,657 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 12:10:01,657 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 12:10:01,657 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:10:01,657 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15241.53 MB 2025-02-14 12:10:01,657 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16914.60 MB 2025-02-14 12:10:01,658 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1673.07 MB 2025-02-14 12:10:01,658 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17955.82 MB 2025-02-14 12:10:01,658 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20260.59 MB 2025-02-14 12:10:01,658 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2304.77 MB 2025-02-14 12:10:01,658 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19159.99 MB 2025-02-14 12:10:01,727 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 12:10:01,727 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 12:10:01,727 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 12:10:01,727 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:10:01,727 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17535.68 MB 2025-02-14 12:10:01,727 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17846.32 MB 2025-02-14 12:10:01,727 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 310.64 MB 2025-02-14 12:10:01,727 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20260.59 MB 2025-02-14 12:10:01,727 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20426.26 MB 2025-02-14 12:10:01,728 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 165.68 MB 2025-02-14 12:10:01,728 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18141.25 MB 2025-02-14 12:10:01,737 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 12:10:01,737 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 12:10:01,737 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:10:01,737 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:10:01,737 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18013.54 MB 2025-02-14 12:10:01,738 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18217.52 MB 2025-02-14 12:10:01,738 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 203.97 MB 2025-02-14 12:10:01,738 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20426.26 MB 2025-02-14 12:10:01,738 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20430.45 MB 2025-02-14 12:10:01,738 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-14 12:10:01,738 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18234.96 MB 2025-02-14 12:10:01,739 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 12:10:01,739 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 12:10:01,739 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.71 seconds 2025-02-14 12:10:01,739 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:10:01,739 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13585.39 MB 2025-02-14 12:10:01,739 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18418.39 MB 2025-02-14 12:10:01,739 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4833.00 MB 2025-02-14 12:10:01,739 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52674.17 MB 2025-02-14 12:10:01,739 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20430.45 MB 2025-02-14 12:10:01,739 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32243.71 MB 2025-02-14 12:10:01,739 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18418.39 MB 2025-02-14 12:10:02,007 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 12:10:02,007 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 12:10:02,007 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 12:10:02,007 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:10:02,007 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18418.39 MB 2025-02-14 12:10:02,007 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17463.67 MB 2025-02-14 12:10:02,007 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -954.72 MB 2025-02-14 12:10:02,007 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20430.45 MB 2025-02-14 12:10:02,007 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20430.45 MB 2025-02-14 12:10:02,007 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:10:02,007 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19120.97 MB 2025-02-14 12:10:02,028 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8154, cut from 8156 2025-02-14 12:10:02,028 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 12:10:02,034 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 12:10:02,035 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 12:10:02,035 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 12:10:02,035 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:10:02,035 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17463.67 MB 2025-02-14 12:10:02,035 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25894.35 MB 2025-02-14 12:10:02,035 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8430.68 MB 2025-02-14 12:10:02,035 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20430.45 MB 2025-02-14 12:10:02,035 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30909.92 MB 2025-02-14 12:10:02,035 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10479.47 MB 2025-02-14 12:10:02,035 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25894.35 MB 2025-02-14 12:10:02,186 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7946] 2025-02-14 12:10:02,187 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:10:02,187 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 12:10:02,188 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:10:02,188 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 12:10:02,193 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 12:10:02,194 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:10:02,194 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 12:10:02,194 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 12:10:39,789 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:10:39,790 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 12:10:39,798 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 12:10:39,805 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:10:39,805 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1295, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 12:10:39,807 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:10:39,807 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1295, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 12:10:59,829 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 12:10:59,829 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 12:10:59,829 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.01 seconds 2025-02-14 12:10:59,829 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:10:59,829 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21992.47 MB 2025-02-14 12:10:59,829 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26575.41 MB 2025-02-14 12:10:59,829 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4582.93 MB 2025-02-14 12:10:59,829 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43482.35 MB 2025-02-14 12:10:59,829 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38256.25 MB 2025-02-14 12:10:59,829 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5226.10 MB 2025-02-14 12:10:59,829 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35540.71 MB 2025-02-14 12:10:59,905 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 12:10:59,905 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 12:10:59,905 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 12:10:59,905 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:10:59,905 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26575.41 MB 2025-02-14 12:10:59,905 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22510.15 MB 2025-02-14 12:10:59,905 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4065.25 MB 2025-02-14 12:10:59,905 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38256.25 MB 2025-02-14 12:10:59,905 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47297.07 MB 2025-02-14 12:10:59,905 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9040.82 MB 2025-02-14 12:10:59,905 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40222.37 MB 2025-02-14 12:11:01,824 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 12:11:01,824 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 12:11:01,824 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 12:11:01,824 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:11:01,824 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22510.15 MB 2025-02-14 12:11:01,824 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23040.99 MB 2025-02-14 12:11:01,824 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 12:11:01,824 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47297.07 MB 2025-02-14 12:11:01,824 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33671.87 MB 2025-02-14 12:11:01,824 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13625.20 MB 2025-02-14 12:11:01,824 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27020.33 MB 2025-02-14 12:11:01,837 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 12:11:01,837 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 12:11:01,837 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:11:01,837 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:11:01,837 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23040.99 MB 2025-02-14 12:11:01,837 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24930.53 MB 2025-02-14 12:11:01,837 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 12:11:01,837 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33671.87 MB 2025-02-14 12:11:01,837 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33671.87 MB 2025-02-14 12:11:01,837 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:11:01,837 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26347.96 MB 2025-02-14 12:11:02,049 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 12:11:02,049 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 12:11:02,049 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 12:11:02,049 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:11:02,049 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24930.53 MB 2025-02-14 12:11:02,049 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27172.38 MB 2025-02-14 12:11:02,049 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 12:11:02,049 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33671.87 MB 2025-02-14 12:11:02,049 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35559.31 MB 2025-02-14 12:11:02,049 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 12:11:02,049 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32716.66 MB 2025-02-14 12:11:02,050 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 12:11:02,050 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 12:11:02,050 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 12:11:02,050 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:11:02,050 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23040.99 MB 2025-02-14 12:11:02,050 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27172.38 MB 2025-02-14 12:11:02,050 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 12:11:02,050 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33671.87 MB 2025-02-14 12:11:02,050 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35559.31 MB 2025-02-14 12:11:02,050 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 12:11:02,050 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32716.66 MB 2025-02-14 12:11:02,219 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 12:11:02,219 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 12:11:02,219 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 12:11:02,219 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:11:02,219 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28705.93 MB 2025-02-14 12:11:02,219 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29472.93 MB 2025-02-14 12:11:02,219 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 12:11:02,219 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35559.31 MB 2025-02-14 12:11:02,219 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35974.55 MB 2025-02-14 12:11:02,219 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 12:11:02,219 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30180.72 MB 2025-02-14 12:11:02,238 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 12:11:02,238 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 12:11:02,238 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:11:02,238 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:11:02,238 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29885.82 MB 2025-02-14 12:11:02,238 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30113.47 MB 2025-02-14 12:11:02,238 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.66 MB 2025-02-14 12:11:02,238 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35974.55 MB 2025-02-14 12:11:02,238 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35974.55 MB 2025-02-14 12:11:02,238 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:11:02,238 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30343.17 MB 2025-02-14 12:11:02,239 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 12:11:02,239 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 12:11:02,239 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.43 seconds 2025-02-14 12:11:02,240 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:11:02,240 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17480.59 MB 2025-02-14 12:11:02,240 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30313.88 MB 2025-02-14 12:11:02,240 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12833.29 MB 2025-02-14 12:11:02,240 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43482.35 MB 2025-02-14 12:11:02,240 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35974.55 MB 2025-02-14 12:11:02,240 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7507.80 MB 2025-02-14 12:11:02,240 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30343.17 MB 2025-02-14 12:11:02,509 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 12:11:02,509 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 12:11:02,509 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 12:11:02,509 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:11:02,509 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30313.88 MB 2025-02-14 12:11:02,509 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22474.69 MB 2025-02-14 12:11:02,509 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7839.19 MB 2025-02-14 12:11:02,509 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35974.55 MB 2025-02-14 12:11:02,509 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35974.55 MB 2025-02-14 12:11:02,509 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:11:02,509 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32817.26 MB 2025-02-14 12:11:02,526 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8135, cut from 8137 2025-02-14 12:11:02,527 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 12:11:02,533 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 12:11:02,533 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 12:11:02,533 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:11:02,533 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:11:02,533 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22474.69 MB 2025-02-14 12:11:02,533 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30885.52 MB 2025-02-14 12:11:02,533 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8410.82 MB 2025-02-14 12:11:02,533 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35974.55 MB 2025-02-14 12:11:02,533 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44337.99 MB 2025-02-14 12:11:02,533 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8363.44 MB 2025-02-14 12:11:02,533 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30885.52 MB 2025-02-14 12:11:02,695 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7927] 2025-02-14 12:11:02,697 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:11:02,697 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 12:11:02,698 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:11:02,698 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 12:11:02,703 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 12:11:02,704 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:11:02,704 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 12:11:02,704 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 12:11:09,260 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:11:09,260 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 12:11:09,266 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 12:11:09,269 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:11:09,269 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 658, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 12:11:09,270 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:11:09,270 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 658, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 12:11:19,545 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 12:11:19,545 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 12:11:19,546 - resource_logging.py:150 - __exit__ - DEBUG - Time: 10.27 seconds 2025-02-14 12:11:19,546 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:11:19,546 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17553.76 MB 2025-02-14 12:11:19,546 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19882.38 MB 2025-02-14 12:11:19,546 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2328.63 MB 2025-02-14 12:11:19,546 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52701.43 MB 2025-02-14 12:11:19,546 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25283.26 MB 2025-02-14 12:11:19,546 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27418.17 MB 2025-02-14 12:11:19,546 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28837.07 MB 2025-02-14 12:11:19,602 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 12:11:19,602 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 12:11:19,602 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-14 12:11:19,602 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:11:19,602 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19882.38 MB 2025-02-14 12:11:19,602 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19198.59 MB 2025-02-14 12:11:19,602 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -683.79 MB 2025-02-14 12:11:19,602 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25283.26 MB 2025-02-14 12:11:19,602 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32514.24 MB 2025-02-14 12:11:19,602 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7230.98 MB 2025-02-14 12:11:19,602 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28511.30 MB 2025-02-14 12:11:21,535 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 12:11:21,535 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 12:11:21,535 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 12:11:21,535 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:11:21,535 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19198.59 MB 2025-02-14 12:11:21,535 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19729.43 MB 2025-02-14 12:11:21,535 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 12:11:21,535 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32514.24 MB 2025-02-14 12:11:21,535 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26698.84 MB 2025-02-14 12:11:21,535 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5815.40 MB 2025-02-14 12:11:21,535 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23708.76 MB 2025-02-14 12:11:21,549 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 12:11:21,549 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 12:11:21,549 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:11:21,549 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:11:21,549 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19729.43 MB 2025-02-14 12:11:21,549 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21618.96 MB 2025-02-14 12:11:21,549 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 12:11:21,549 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26698.84 MB 2025-02-14 12:11:21,549 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26698.84 MB 2025-02-14 12:11:21,549 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:11:21,549 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23036.39 MB 2025-02-14 12:11:21,759 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 12:11:21,759 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 12:11:21,759 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 12:11:21,760 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:11:21,760 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21618.96 MB 2025-02-14 12:11:21,760 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23860.82 MB 2025-02-14 12:11:21,760 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 12:11:21,760 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26698.84 MB 2025-02-14 12:11:21,760 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32361.15 MB 2025-02-14 12:11:21,760 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 12:11:21,760 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29405.10 MB 2025-02-14 12:11:21,760 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 12:11:21,760 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 12:11:21,760 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 12:11:21,760 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:11:21,760 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19729.43 MB 2025-02-14 12:11:21,760 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23860.82 MB 2025-02-14 12:11:21,760 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 12:11:21,760 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26698.84 MB 2025-02-14 12:11:21,760 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32361.15 MB 2025-02-14 12:11:21,760 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 12:11:21,760 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29405.10 MB 2025-02-14 12:11:21,934 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 12:11:21,934 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 12:11:21,934 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 12:11:21,934 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:11:21,934 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25394.36 MB 2025-02-14 12:11:21,934 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26161.36 MB 2025-02-14 12:11:21,934 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 12:11:21,934 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32361.15 MB 2025-02-14 12:11:21,934 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32774.29 MB 2025-02-14 12:11:21,934 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 12:11:21,935 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26869.15 MB 2025-02-14 12:11:21,954 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 12:11:21,954 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 12:11:21,954 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:11:21,954 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:11:21,954 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26574.25 MB 2025-02-14 12:11:21,954 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26803.41 MB 2025-02-14 12:11:21,954 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.15 MB 2025-02-14 12:11:21,954 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32774.29 MB 2025-02-14 12:11:21,954 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32774.29 MB 2025-02-14 12:11:21,954 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:11:21,954 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26993.95 MB 2025-02-14 12:11:21,955 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 12:11:21,955 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 12:11:21,955 - resource_logging.py:150 - __exit__ - DEBUG - Time: 12.68 seconds 2025-02-14 12:11:21,955 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:11:21,955 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15261.23 MB 2025-02-14 12:11:21,955 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27003.72 MB 2025-02-14 12:11:21,955 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11742.49 MB 2025-02-14 12:11:21,955 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52701.43 MB 2025-02-14 12:11:21,955 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32774.29 MB 2025-02-14 12:11:21,956 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19927.14 MB 2025-02-14 12:11:21,956 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27003.72 MB 2025-02-14 12:11:22,223 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 12:11:22,223 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 12:11:22,223 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 12:11:22,223 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:11:22,223 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27003.72 MB 2025-02-14 12:11:22,223 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20253.81 MB 2025-02-14 12:11:22,223 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6749.90 MB 2025-02-14 12:11:22,223 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32774.29 MB 2025-02-14 12:11:22,223 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32774.29 MB 2025-02-14 12:11:22,223 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:11:22,223 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29505.86 MB 2025-02-14 12:11:22,241 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8131, cut from 8133 2025-02-14 12:11:22,241 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 12:11:22,247 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 12:11:22,247 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 12:11:22,247 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:11:22,247 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:11:22,247 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20253.81 MB 2025-02-14 12:11:22,247 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28661.55 MB 2025-02-14 12:11:22,247 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8407.74 MB 2025-02-14 12:11:22,248 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32774.29 MB 2025-02-14 12:11:22,248 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41133.54 MB 2025-02-14 12:11:22,248 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8359.25 MB 2025-02-14 12:11:22,248 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28661.55 MB 2025-02-14 12:11:22,408 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7923] 2025-02-14 12:11:22,410 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:11:22,410 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 12:11:22,411 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:11:22,411 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 12:11:22,416 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 12:11:22,417 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:11:22,417 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 12:11:22,417 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 12:12:13,215 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:12:13,215 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 12:12:13,220 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 12:12:13,223 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:12:13,223 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 86, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 12:12:13,224 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:12:13,224 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 86, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 12:12:14,584 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 12:12:14,584 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 12:12:14,585 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.36 seconds 2025-02-14 12:12:14,585 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:12:14,585 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13567.97 MB 2025-02-14 12:12:14,585 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 13872.32 MB 2025-02-14 12:12:14,585 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 304.35 MB 2025-02-14 12:12:14,585 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49492.79 MB 2025-02-14 12:12:14,585 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 16909.34 MB 2025-02-14 12:12:14,585 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32583.45 MB 2025-02-14 12:12:14,585 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22812.85 MB 2025-02-14 12:12:14,587 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 12:12:14,587 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 12:12:14,587 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 12:12:14,588 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:12:14,588 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13872.32 MB 2025-02-14 12:12:14,588 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14019.77 MB 2025-02-14 12:12:14,588 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 147.46 MB 2025-02-14 12:12:14,588 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 16909.34 MB 2025-02-14 12:12:14,588 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 16909.34 MB 2025-02-14 12:12:14,588 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:12:14,588 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14476.36 MB 2025-02-14 12:12:15,008 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 12:12:15,008 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 12:12:15,008 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.42 seconds 2025-02-14 12:12:15,008 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:12:15,008 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14019.77 MB 2025-02-14 12:12:15,008 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14133.90 MB 2025-02-14 12:12:15,008 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 114.13 MB 2025-02-14 12:12:15,008 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 16909.34 MB 2025-02-14 12:12:15,008 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 16909.34 MB 2025-02-14 12:12:15,008 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:12:15,008 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18106.31 MB 2025-02-14 12:12:15,014 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 12:12:15,014 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 12:12:15,014 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 12:12:15,014 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:12:15,014 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14133.84 MB 2025-02-14 12:12:15,014 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14539.99 MB 2025-02-14 12:12:15,014 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 406.15 MB 2025-02-14 12:12:15,014 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 16909.34 MB 2025-02-14 12:12:15,014 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 16909.34 MB 2025-02-14 12:12:15,014 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:12:15,014 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14844.74 MB 2025-02-14 12:12:15,102 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 12:12:15,102 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 12:12:15,102 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 12:12:15,102 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:12:15,102 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14539.99 MB 2025-02-14 12:12:15,102 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15033.31 MB 2025-02-14 12:12:15,102 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 493.32 MB 2025-02-14 12:12:15,102 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 16909.34 MB 2025-02-14 12:12:15,102 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 16909.34 MB 2025-02-14 12:12:15,102 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:12:15,102 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16214.80 MB 2025-02-14 12:12:15,103 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 12:12:15,103 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 12:12:15,103 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 12:12:15,103 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:12:15,103 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14133.84 MB 2025-02-14 12:12:15,103 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15033.31 MB 2025-02-14 12:12:15,103 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 899.47 MB 2025-02-14 12:12:15,103 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 16909.34 MB 2025-02-14 12:12:15,103 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 16909.34 MB 2025-02-14 12:12:15,103 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:12:15,103 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16214.80 MB 2025-02-14 12:12:15,148 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 12:12:15,149 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 12:12:15,149 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-14 12:12:15,149 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:12:15,149 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15509.93 MB 2025-02-14 12:12:15,149 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15717.90 MB 2025-02-14 12:12:15,149 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 207.96 MB 2025-02-14 12:12:15,149 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 16909.34 MB 2025-02-14 12:12:15,149 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17039.36 MB 2025-02-14 12:12:15,149 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 130.02 MB 2025-02-14 12:12:15,149 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15870.07 MB 2025-02-14 12:12:15,154 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 12:12:15,154 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 12:12:15,154 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 12:12:15,154 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:12:15,154 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15848.95 MB 2025-02-14 12:12:15,154 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16053.95 MB 2025-02-14 12:12:15,154 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.00 MB 2025-02-14 12:12:15,154 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17039.36 MB 2025-02-14 12:12:15,154 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17039.36 MB 2025-02-14 12:12:15,154 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:12:15,154 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16053.95 MB 2025-02-14 12:12:15,155 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 12:12:15,155 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 12:12:15,155 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 12:12:15,155 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:12:15,155 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13268.34 MB 2025-02-14 12:12:15,155 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16237.44 MB 2025-02-14 12:12:15,155 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2969.10 MB 2025-02-14 12:12:15,155 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49492.79 MB 2025-02-14 12:12:15,155 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17039.36 MB 2025-02-14 12:12:15,155 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32453.43 MB 2025-02-14 12:12:15,155 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16237.44 MB 2025-02-14 12:12:15,401 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 12:12:15,401 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 12:12:15,401 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.24 seconds 2025-02-14 12:12:15,401 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:12:15,401 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13768.29 MB 2025-02-14 12:12:15,401 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16518.74 MB 2025-02-14 12:12:15,401 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2750.45 MB 2025-02-14 12:12:15,401 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17039.36 MB 2025-02-14 12:12:15,401 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17905.48 MB 2025-02-14 12:12:15,401 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 866.12 MB 2025-02-14 12:12:15,401 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16793.75 MB 2025-02-14 12:12:15,417 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 7447, cut from 7449 2025-02-14 12:12:15,418 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 12:12:15,423 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 12:12:15,423 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 12:12:15,423 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:12:15,423 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:12:15,423 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16518.74 MB 2025-02-14 12:12:15,423 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24219.56 MB 2025-02-14 12:12:15,423 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7700.82 MB 2025-02-14 12:12:15,423 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17905.48 MB 2025-02-14 12:12:15,423 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27476.89 MB 2025-02-14 12:12:15,423 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9571.40 MB 2025-02-14 12:12:15,423 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24219.56 MB 2025-02-14 12:12:15,573 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7239] 2025-02-14 12:12:15,574 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:12:15,574 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 12:12:15,575 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:12:15,575 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 12:12:15,580 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 12:12:15,581 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:12:15,581 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 12:12:15,581 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 12:13:10,048 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:13:10,048 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 12:13:10,053 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 12:13:10,056 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:13:10,056 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1081, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 12:13:10,057 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:13:10,057 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1081, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 12:13:26,613 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 12:13:26,613 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 12:13:26,613 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.55 seconds 2025-02-14 12:13:26,613 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:13:26,613 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20501.29 MB 2025-02-14 12:13:26,613 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24326.89 MB 2025-02-14 12:13:26,613 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3825.60 MB 2025-02-14 12:13:26,613 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38960.89 MB 2025-02-14 12:13:26,613 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30305.94 MB 2025-02-14 12:13:26,613 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8654.95 MB 2025-02-14 12:13:26,613 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33144.36 MB 2025-02-14 12:13:26,678 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 12:13:26,678 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 12:13:26,678 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 12:13:26,678 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:13:26,678 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24326.89 MB 2025-02-14 12:13:26,678 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21397.63 MB 2025-02-14 12:13:26,678 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2929.25 MB 2025-02-14 12:13:26,678 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30305.94 MB 2025-02-14 12:13:26,678 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40187.72 MB 2025-02-14 12:13:26,678 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9881.78 MB 2025-02-14 12:13:26,678 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35698.44 MB 2025-02-14 12:13:28,609 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 12:13:28,610 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 12:13:28,610 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 12:13:28,610 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:13:28,610 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21397.63 MB 2025-02-14 12:13:28,610 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21928.47 MB 2025-02-14 12:13:28,610 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 12:13:28,610 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40187.72 MB 2025-02-14 12:13:28,610 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27894.22 MB 2025-02-14 12:13:28,610 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12293.51 MB 2025-02-14 12:13:28,610 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25907.81 MB 2025-02-14 12:13:28,623 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 12:13:28,623 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 12:13:28,623 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:13:28,623 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:13:28,623 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21928.47 MB 2025-02-14 12:13:28,623 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23817.81 MB 2025-02-14 12:13:28,623 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.34 MB 2025-02-14 12:13:28,623 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27894.22 MB 2025-02-14 12:13:28,624 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27894.22 MB 2025-02-14 12:13:28,624 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:13:28,624 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25235.24 MB 2025-02-14 12:13:28,834 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 12:13:28,834 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 12:13:28,834 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 12:13:28,834 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:13:28,834 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23817.81 MB 2025-02-14 12:13:28,834 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26059.67 MB 2025-02-14 12:13:28,834 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 12:13:28,834 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27894.22 MB 2025-02-14 12:13:28,834 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33556.53 MB 2025-02-14 12:13:28,834 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 12:13:28,834 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31603.95 MB 2025-02-14 12:13:28,835 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 12:13:28,835 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 12:13:28,835 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 12:13:28,835 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:13:28,835 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21928.47 MB 2025-02-14 12:13:28,835 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26059.67 MB 2025-02-14 12:13:28,835 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.19 MB 2025-02-14 12:13:28,835 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27894.22 MB 2025-02-14 12:13:28,835 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33556.53 MB 2025-02-14 12:13:28,835 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 12:13:28,835 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31603.95 MB 2025-02-14 12:13:29,003 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 12:13:29,003 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 12:13:29,003 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 12:13:29,004 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:13:29,004 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27593.21 MB 2025-02-14 12:13:29,004 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28360.21 MB 2025-02-14 12:13:29,004 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 12:13:29,004 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33556.53 MB 2025-02-14 12:13:29,004 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33971.77 MB 2025-02-14 12:13:29,004 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 12:13:29,004 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29068.00 MB 2025-02-14 12:13:29,023 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 12:13:29,023 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 12:13:29,023 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:13:29,023 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:13:29,023 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28773.10 MB 2025-02-14 12:13:29,023 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29002.89 MB 2025-02-14 12:13:29,023 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.78 MB 2025-02-14 12:13:29,023 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33971.77 MB 2025-02-14 12:13:29,023 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33971.77 MB 2025-02-14 12:13:29,023 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:13:29,023 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29202.89 MB 2025-02-14 12:13:29,024 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 12:13:29,024 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 12:13:29,024 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.96 seconds 2025-02-14 12:13:29,024 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:13:29,024 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16735.00 MB 2025-02-14 12:13:29,024 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29203.96 MB 2025-02-14 12:13:29,024 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12468.96 MB 2025-02-14 12:13:29,024 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38960.89 MB 2025-02-14 12:13:29,024 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33971.77 MB 2025-02-14 12:13:29,024 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4989.12 MB 2025-02-14 12:13:29,024 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29203.96 MB 2025-02-14 12:13:29,293 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 12:13:29,293 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 12:13:29,293 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 12:13:29,293 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:13:29,293 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29203.96 MB 2025-02-14 12:13:29,293 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21739.39 MB 2025-02-14 12:13:29,293 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7464.57 MB 2025-02-14 12:13:29,293 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33971.77 MB 2025-02-14 12:13:29,293 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33971.77 MB 2025-02-14 12:13:29,293 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:13:29,293 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31715.63 MB 2025-02-14 12:13:29,311 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 12:13:29,311 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 12:13:29,317 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 12:13:29,317 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 12:13:29,317 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:13:29,317 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:13:29,317 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21739.39 MB 2025-02-14 12:13:29,317 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30178.41 MB 2025-02-14 12:13:29,317 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 12:13:29,317 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33971.77 MB 2025-02-14 12:13:29,317 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42362.47 MB 2025-02-14 12:13:29,317 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 12:13:29,317 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30178.41 MB 2025-02-14 12:13:29,480 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 12:13:29,481 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:13:29,481 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 12:13:29,482 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:13:29,482 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 12:13:29,487 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 12:13:29,488 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:13:29,488 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 12:13:29,488 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 12:14:26,434 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:14:26,434 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 12:14:26,440 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 12:14:26,443 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:14:26,443 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1386, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 12:14:26,444 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:14:26,444 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1386, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 12:14:47,773 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 12:14:47,773 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 12:14:47,773 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.32 seconds 2025-02-14 12:14:47,773 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:14:47,773 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22626.58 MB 2025-02-14 12:14:47,773 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27531.81 MB 2025-02-14 12:14:47,773 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4905.24 MB 2025-02-14 12:14:47,773 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54947.48 MB 2025-02-14 12:14:47,773 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37857.79 MB 2025-02-14 12:14:47,773 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17089.69 MB 2025-02-14 12:14:47,773 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36401.30 MB 2025-02-14 12:14:47,852 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 12:14:47,852 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 12:14:47,852 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 12:14:47,852 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:14:47,852 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27531.81 MB 2025-02-14 12:14:47,852 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22983.23 MB 2025-02-14 12:14:47,852 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4548.58 MB 2025-02-14 12:14:47,852 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37857.79 MB 2025-02-14 12:14:47,852 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46986.69 MB 2025-02-14 12:14:47,852 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9128.90 MB 2025-02-14 12:14:47,852 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41288.12 MB 2025-02-14 12:14:49,776 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 12:14:49,776 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 12:14:49,776 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 12:14:49,776 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:14:49,776 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22983.23 MB 2025-02-14 12:14:49,776 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23514.07 MB 2025-02-14 12:14:49,776 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 12:14:49,777 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46986.69 MB 2025-02-14 12:14:49,777 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29125.25 MB 2025-02-14 12:14:49,777 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17861.44 MB 2025-02-14 12:14:49,777 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27493.41 MB 2025-02-14 12:14:49,792 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 12:14:49,792 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 12:14:49,792 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:14:49,792 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:14:49,792 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23514.07 MB 2025-02-14 12:14:49,792 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25403.61 MB 2025-02-14 12:14:49,792 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 12:14:49,792 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29125.25 MB 2025-02-14 12:14:49,792 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30068.97 MB 2025-02-14 12:14:49,792 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 12:14:49,792 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26821.04 MB 2025-02-14 12:14:50,005 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 12:14:50,005 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 12:14:50,005 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 12:14:50,005 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:14:50,005 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25403.61 MB 2025-02-14 12:14:50,005 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27645.46 MB 2025-02-14 12:14:50,005 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 12:14:50,005 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30068.97 MB 2025-02-14 12:14:50,005 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36203.13 MB 2025-02-14 12:14:50,005 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 12:14:50,005 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33189.75 MB 2025-02-14 12:14:50,006 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 12:14:50,006 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 12:14:50,006 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 12:14:50,006 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:14:50,006 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23514.07 MB 2025-02-14 12:14:50,006 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27645.46 MB 2025-02-14 12:14:50,006 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 12:14:50,006 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29125.25 MB 2025-02-14 12:14:50,006 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36203.13 MB 2025-02-14 12:14:50,006 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7077.89 MB 2025-02-14 12:14:50,006 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33189.75 MB 2025-02-14 12:14:50,176 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 12:14:50,176 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 12:14:50,176 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 12:14:50,176 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:14:50,176 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29179.01 MB 2025-02-14 12:14:50,176 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29946.01 MB 2025-02-14 12:14:50,176 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 12:14:50,176 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36203.13 MB 2025-02-14 12:14:50,176 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36618.37 MB 2025-02-14 12:14:50,176 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 12:14:50,176 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30653.80 MB 2025-02-14 12:14:50,196 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 12:14:50,196 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 12:14:50,196 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:14:50,196 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:14:50,196 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30358.90 MB 2025-02-14 12:14:50,196 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30587.22 MB 2025-02-14 12:14:50,196 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.32 MB 2025-02-14 12:14:50,196 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36618.37 MB 2025-02-14 12:14:50,196 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36618.37 MB 2025-02-14 12:14:50,196 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:14:50,196 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30801.90 MB 2025-02-14 12:14:50,197 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 12:14:50,197 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 12:14:50,197 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.75 seconds 2025-02-14 12:14:50,197 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:14:50,197 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17797.64 MB 2025-02-14 12:14:50,197 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30787.46 MB 2025-02-14 12:14:50,197 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12989.81 MB 2025-02-14 12:14:50,197 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54947.48 MB 2025-02-14 12:14:50,197 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36618.37 MB 2025-02-14 12:14:50,197 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18329.11 MB 2025-02-14 12:14:50,197 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30801.90 MB 2025-02-14 12:14:50,466 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 12:14:50,466 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 12:14:50,466 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 12:14:50,466 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:14:50,466 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30787.46 MB 2025-02-14 12:14:50,466 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22789.08 MB 2025-02-14 12:14:50,466 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7998.38 MB 2025-02-14 12:14:50,466 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36618.37 MB 2025-02-14 12:14:50,466 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36618.37 MB 2025-02-14 12:14:50,466 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:14:50,466 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33288.68 MB 2025-02-14 12:14:50,484 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8128, cut from 8130 2025-02-14 12:14:50,485 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 12:14:50,491 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 12:14:50,491 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 12:14:50,491 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:14:50,491 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:14:50,491 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22789.08 MB 2025-02-14 12:14:50,491 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31194.16 MB 2025-02-14 12:14:50,491 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8405.08 MB 2025-02-14 12:14:50,491 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36618.37 MB 2025-02-14 12:14:50,491 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44973.42 MB 2025-02-14 12:14:50,491 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8355.05 MB 2025-02-14 12:14:50,491 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31194.16 MB 2025-02-14 12:14:50,654 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7920] 2025-02-14 12:14:50,655 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:14:50,655 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 12:14:50,656 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:14:50,656 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 12:14:50,661 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 12:14:50,662 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:14:50,662 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 12:14:50,662 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 12:15:27,421 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:15:27,421 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 12:15:27,426 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 12:15:27,430 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:15:27,430 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1209, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 12:15:27,431 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:15:27,431 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1209, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 12:15:46,159 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 12:15:46,159 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 12:15:46,159 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.72 seconds 2025-02-14 12:15:46,159 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:15:46,159 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21393.21 MB 2025-02-14 12:15:46,159 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25671.80 MB 2025-02-14 12:15:46,159 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4278.58 MB 2025-02-14 12:15:46,159 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53328.48 MB 2025-02-14 12:15:46,159 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33386.66 MB 2025-02-14 12:15:46,159 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19941.82 MB 2025-02-14 12:15:46,159 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34488.46 MB 2025-02-14 12:15:46,234 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 12:15:46,234 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 12:15:46,234 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 12:15:46,234 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:15:46,234 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25671.80 MB 2025-02-14 12:15:46,234 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22063.07 MB 2025-02-14 12:15:46,234 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3608.73 MB 2025-02-14 12:15:46,234 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33386.66 MB 2025-02-14 12:15:46,234 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42117.10 MB 2025-02-14 12:15:46,234 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8730.44 MB 2025-02-14 12:15:46,234 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38019.24 MB 2025-02-14 12:15:48,186 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 12:15:48,186 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 12:15:48,186 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.95 seconds 2025-02-14 12:15:48,186 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:15:48,186 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22063.07 MB 2025-02-14 12:15:48,186 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22593.91 MB 2025-02-14 12:15:48,186 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 12:15:48,186 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42117.10 MB 2025-02-14 12:15:48,186 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25279.07 MB 2025-02-14 12:15:48,186 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16838.03 MB 2025-02-14 12:15:48,186 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26574.28 MB 2025-02-14 12:15:48,200 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 12:15:48,200 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 12:15:48,200 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:15:48,200 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:15:48,200 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22593.91 MB 2025-02-14 12:15:48,200 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24483.44 MB 2025-02-14 12:15:48,200 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 12:15:48,200 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25279.07 MB 2025-02-14 12:15:48,200 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28110.23 MB 2025-02-14 12:15:48,200 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-14 12:15:48,200 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25900.87 MB 2025-02-14 12:15:48,414 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 12:15:48,414 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 12:15:48,414 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 12:15:48,414 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:15:48,414 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24483.44 MB 2025-02-14 12:15:48,414 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26725.30 MB 2025-02-14 12:15:48,414 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 12:15:48,414 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28110.23 MB 2025-02-14 12:15:48,414 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34244.40 MB 2025-02-14 12:15:48,414 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 12:15:48,415 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32269.58 MB 2025-02-14 12:15:48,415 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 12:15:48,415 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 12:15:48,415 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 12:15:48,415 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:15:48,415 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22593.91 MB 2025-02-14 12:15:48,415 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26725.30 MB 2025-02-14 12:15:48,415 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 12:15:48,415 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25279.07 MB 2025-02-14 12:15:48,415 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34244.40 MB 2025-02-14 12:15:48,415 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8965.32 MB 2025-02-14 12:15:48,415 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32269.58 MB 2025-02-14 12:15:48,594 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 12:15:48,594 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 12:15:48,594 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 12:15:48,594 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:15:48,594 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28258.84 MB 2025-02-14 12:15:48,594 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29025.84 MB 2025-02-14 12:15:48,594 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 12:15:48,594 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34244.40 MB 2025-02-14 12:15:48,594 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34661.73 MB 2025-02-14 12:15:48,594 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 12:15:48,594 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29733.63 MB 2025-02-14 12:15:48,615 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 12:15:48,615 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 12:15:48,615 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:15:48,615 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:15:48,615 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29438.73 MB 2025-02-14 12:15:48,615 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29666.42 MB 2025-02-14 12:15:48,615 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.69 MB 2025-02-14 12:15:48,615 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34661.73 MB 2025-02-14 12:15:48,615 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34661.73 MB 2025-02-14 12:15:48,615 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:15:48,615 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29882.87 MB 2025-02-14 12:15:48,616 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 12:15:48,616 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 12:15:48,616 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.18 seconds 2025-02-14 12:15:48,616 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:15:48,616 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17180.96 MB 2025-02-14 12:15:48,616 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29866.29 MB 2025-02-14 12:15:48,616 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12685.33 MB 2025-02-14 12:15:48,616 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53328.48 MB 2025-02-14 12:15:48,616 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34661.73 MB 2025-02-14 12:15:48,616 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18666.75 MB 2025-02-14 12:15:48,616 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29882.87 MB 2025-02-14 12:15:48,887 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 12:15:48,887 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 12:15:48,887 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 12:15:48,887 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:15:48,887 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29866.29 MB 2025-02-14 12:15:48,887 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22166.68 MB 2025-02-14 12:15:48,887 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7699.61 MB 2025-02-14 12:15:48,887 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34661.73 MB 2025-02-14 12:15:48,887 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34661.73 MB 2025-02-14 12:15:48,887 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:15:48,887 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32362.91 MB 2025-02-14 12:15:48,905 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8113, cut from 8115 2025-02-14 12:15:48,905 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 12:15:48,911 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 12:15:48,911 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 12:15:48,911 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:15:48,911 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:15:48,911 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22166.68 MB 2025-02-14 12:15:48,911 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30555.10 MB 2025-02-14 12:15:48,911 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8388.42 MB 2025-02-14 12:15:48,911 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34661.73 MB 2025-02-14 12:15:48,912 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43002.10 MB 2025-02-14 12:15:48,912 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8340.37 MB 2025-02-14 12:15:48,912 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30555.10 MB 2025-02-14 12:15:49,081 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7905] 2025-02-14 12:15:49,083 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:15:49,083 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 12:15:49,084 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:15:49,084 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 12:15:49,089 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 12:15:49,090 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:15:49,090 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 12:15:49,090 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 12:15:55,840 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:15:55,840 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 12:15:55,845 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 12:15:55,849 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:15:55,849 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 808, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 12:15:55,850 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:15:55,850 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 808, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 12:16:08,500 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 12:16:08,500 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 12:16:08,500 - resource_logging.py:150 - __exit__ - DEBUG - Time: 12.64 seconds 2025-02-14 12:16:08,500 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:16:08,500 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18598.98 MB 2025-02-14 12:16:08,500 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21459.49 MB 2025-02-14 12:16:08,500 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2860.52 MB 2025-02-14 12:16:08,500 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55511.61 MB 2025-02-14 12:16:08,500 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28139.59 MB 2025-02-14 12:16:08,500 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27372.03 MB 2025-02-14 12:16:08,500 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30336.08 MB 2025-02-14 12:16:08,551 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 12:16:08,551 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 12:16:08,551 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-14 12:16:08,551 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:16:08,551 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21459.49 MB 2025-02-14 12:16:08,551 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19978.39 MB 2025-02-14 12:16:08,551 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1481.10 MB 2025-02-14 12:16:08,551 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28139.59 MB 2025-02-14 12:16:08,551 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36605.79 MB 2025-02-14 12:16:08,552 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8466.20 MB 2025-02-14 12:16:08,552 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31346.33 MB 2025-02-14 12:16:10,473 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 12:16:10,473 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 12:16:10,473 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 12:16:10,473 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:16:10,473 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19978.39 MB 2025-02-14 12:16:10,473 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20509.23 MB 2025-02-14 12:16:10,473 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 12:16:10,473 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36605.79 MB 2025-02-14 12:16:10,473 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26694.65 MB 2025-02-14 12:16:10,473 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9911.14 MB 2025-02-14 12:16:10,473 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24488.57 MB 2025-02-14 12:16:10,486 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 12:16:10,486 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 12:16:10,486 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:16:10,486 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:16:10,486 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20509.23 MB 2025-02-14 12:16:10,486 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22398.77 MB 2025-02-14 12:16:10,486 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 12:16:10,486 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26694.65 MB 2025-02-14 12:16:10,486 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26694.65 MB 2025-02-14 12:16:10,486 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:16:10,486 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23816.20 MB 2025-02-14 12:16:10,697 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 12:16:10,697 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 12:16:10,697 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 12:16:10,697 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:16:10,697 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22398.77 MB 2025-02-14 12:16:10,697 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24640.62 MB 2025-02-14 12:16:10,697 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 12:16:10,697 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26694.65 MB 2025-02-14 12:16:10,697 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32828.82 MB 2025-02-14 12:16:10,697 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 12:16:10,697 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30184.90 MB 2025-02-14 12:16:10,698 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 12:16:10,698 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 12:16:10,698 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 12:16:10,698 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:16:10,698 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20509.23 MB 2025-02-14 12:16:10,698 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24640.62 MB 2025-02-14 12:16:10,698 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 12:16:10,698 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26694.65 MB 2025-02-14 12:16:10,698 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32828.82 MB 2025-02-14 12:16:10,698 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 12:16:10,698 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30184.90 MB 2025-02-14 12:16:10,867 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 12:16:10,867 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 12:16:10,867 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 12:16:10,867 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:16:10,867 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26174.16 MB 2025-02-14 12:16:10,867 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26941.17 MB 2025-02-14 12:16:10,867 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 12:16:10,867 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32828.82 MB 2025-02-14 12:16:10,867 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33246.15 MB 2025-02-14 12:16:10,867 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 12:16:10,867 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27648.95 MB 2025-02-14 12:16:10,886 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 12:16:10,886 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 12:16:10,886 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:16:10,886 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:16:10,886 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27354.06 MB 2025-02-14 12:16:10,886 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27582.48 MB 2025-02-14 12:16:10,886 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.42 MB 2025-02-14 12:16:10,886 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33246.15 MB 2025-02-14 12:16:10,886 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33246.15 MB 2025-02-14 12:16:10,886 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:16:10,886 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27802.76 MB 2025-02-14 12:16:10,887 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 12:16:10,887 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 12:16:10,887 - resource_logging.py:150 - __exit__ - DEBUG - Time: 15.04 seconds 2025-02-14 12:16:10,887 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:16:10,887 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15783.84 MB 2025-02-14 12:16:10,887 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27783.08 MB 2025-02-14 12:16:10,887 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11999.24 MB 2025-02-14 12:16:10,887 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55511.61 MB 2025-02-14 12:16:10,887 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33246.15 MB 2025-02-14 12:16:10,887 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22265.46 MB 2025-02-14 12:16:10,887 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27802.76 MB 2025-02-14 12:16:11,158 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 12:16:11,158 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 12:16:11,158 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 12:16:11,158 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:16:11,158 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27783.08 MB 2025-02-14 12:16:11,158 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20780.99 MB 2025-02-14 12:16:11,158 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7002.09 MB 2025-02-14 12:16:11,158 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33246.15 MB 2025-02-14 12:16:11,159 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33246.15 MB 2025-02-14 12:16:11,159 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:16:11,159 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30288.91 MB 2025-02-14 12:16:11,176 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8143, cut from 8145 2025-02-14 12:16:11,177 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 12:16:11,183 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 12:16:11,183 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 12:16:11,183 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:16:11,183 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:16:11,183 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20780.99 MB 2025-02-14 12:16:11,183 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29200.07 MB 2025-02-14 12:16:11,183 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8419.08 MB 2025-02-14 12:16:11,183 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33246.15 MB 2025-02-14 12:16:11,183 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43710.94 MB 2025-02-14 12:16:11,183 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10464.79 MB 2025-02-14 12:16:11,183 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29200.07 MB 2025-02-14 12:16:11,346 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7935] 2025-02-14 12:16:11,347 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:16:11,347 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 12:16:11,348 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:16:11,348 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 12:16:11,353 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 12:16:11,354 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:16:11,354 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 12:16:11,354 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 12:16:56,811 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:16:56,811 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 12:16:56,816 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 12:16:56,820 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:16:56,820 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 90, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 12:16:56,821 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:16:56,821 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 90, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 12:16:58,237 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 12:16:58,237 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 12:16:58,237 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.41 seconds 2025-02-14 12:16:58,237 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:16:58,237 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13595.84 MB 2025-02-14 12:16:58,237 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 13914.35 MB 2025-02-14 12:16:58,237 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 318.50 MB 2025-02-14 12:16:58,237 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52082.77 MB 2025-02-14 12:16:58,238 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21084.77 MB 2025-02-14 12:16:58,238 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30998.00 MB 2025-02-14 12:16:58,238 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22840.72 MB 2025-02-14 12:16:58,241 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 12:16:58,241 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 12:16:58,241 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 12:16:58,241 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:16:58,241 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13914.35 MB 2025-02-14 12:16:58,241 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14068.66 MB 2025-02-14 12:16:58,241 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 154.31 MB 2025-02-14 12:16:58,241 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21084.77 MB 2025-02-14 12:16:58,241 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21084.77 MB 2025-02-14 12:16:58,241 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:16:58,241 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14546.48 MB 2025-02-14 12:16:58,680 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 12:16:58,680 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 12:16:58,680 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.44 seconds 2025-02-14 12:16:58,680 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:16:58,680 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14068.66 MB 2025-02-14 12:16:58,680 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14188.10 MB 2025-02-14 12:16:58,680 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 119.44 MB 2025-02-14 12:16:58,680 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21084.77 MB 2025-02-14 12:16:58,680 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21084.77 MB 2025-02-14 12:16:58,680 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:16:58,680 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18154.16 MB 2025-02-14 12:16:58,687 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 12:16:58,687 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 12:16:58,687 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 12:16:58,687 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:16:58,687 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14188.03 MB 2025-02-14 12:16:58,687 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14613.08 MB 2025-02-14 12:16:58,687 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 425.04 MB 2025-02-14 12:16:58,687 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21084.77 MB 2025-02-14 12:16:58,687 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21084.77 MB 2025-02-14 12:16:58,687 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:16:58,687 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14932.00 MB 2025-02-14 12:16:58,778 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 12:16:58,778 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 12:16:58,778 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 12:16:58,778 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:16:58,778 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14613.08 MB 2025-02-14 12:16:58,778 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15129.34 MB 2025-02-14 12:16:58,778 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 516.26 MB 2025-02-14 12:16:58,778 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21084.77 MB 2025-02-14 12:16:58,778 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21084.77 MB 2025-02-14 12:16:58,778 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:16:58,778 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16364.96 MB 2025-02-14 12:16:58,779 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 12:16:58,779 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 12:16:58,779 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 12:16:58,779 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:16:58,779 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14188.03 MB 2025-02-14 12:16:58,779 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15129.34 MB 2025-02-14 12:16:58,779 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 941.31 MB 2025-02-14 12:16:58,779 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21084.77 MB 2025-02-14 12:16:58,779 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21084.77 MB 2025-02-14 12:16:58,779 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:16:58,779 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16364.96 MB 2025-02-14 12:16:58,830 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 12:16:58,830 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 12:16:58,830 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-14 12:16:58,830 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:16:58,830 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15627.74 MB 2025-02-14 12:16:58,830 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15844.55 MB 2025-02-14 12:16:58,830 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 216.81 MB 2025-02-14 12:16:58,830 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21084.77 MB 2025-02-14 12:16:58,831 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21223.18 MB 2025-02-14 12:16:58,831 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 138.41 MB 2025-02-14 12:16:58,831 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16003.81 MB 2025-02-14 12:16:58,836 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 12:16:58,836 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 12:16:58,836 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 12:16:58,836 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:16:58,836 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15981.70 MB 2025-02-14 12:16:58,836 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16196.57 MB 2025-02-14 12:16:58,836 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 214.87 MB 2025-02-14 12:16:58,836 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21223.18 MB 2025-02-14 12:16:58,836 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21223.18 MB 2025-02-14 12:16:58,837 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:16:58,837 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16196.57 MB 2025-02-14 12:16:58,838 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 12:16:58,838 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 12:16:58,838 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.01 seconds 2025-02-14 12:16:58,838 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:16:58,838 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13282.27 MB 2025-02-14 12:16:58,838 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16388.32 MB 2025-02-14 12:16:58,838 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3106.05 MB 2025-02-14 12:16:58,838 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52082.77 MB 2025-02-14 12:16:58,838 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21223.18 MB 2025-02-14 12:16:58,838 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30859.59 MB 2025-02-14 12:16:58,838 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16388.32 MB 2025-02-14 12:16:59,094 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 12:16:59,095 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 12:16:59,095 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.25 seconds 2025-02-14 12:16:59,095 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:16:59,095 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13804.97 MB 2025-02-14 12:16:59,095 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16679.29 MB 2025-02-14 12:16:59,095 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2874.32 MB 2025-02-14 12:16:59,095 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21223.18 MB 2025-02-14 12:16:59,095 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21223.18 MB 2025-02-14 12:16:59,095 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:16:59,095 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16966.69 MB 2025-02-14 12:16:59,112 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 7783, cut from 7785 2025-02-14 12:16:59,112 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2 ('] 2025-02-14 12:16:59,118 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 12:16:59,118 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 12:16:59,118 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:16:59,118 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:16:59,118 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16679.29 MB 2025-02-14 12:16:59,118 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24726.78 MB 2025-02-14 12:16:59,118 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8047.49 MB 2025-02-14 12:16:59,118 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21223.18 MB 2025-02-14 12:16:59,118 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29225.91 MB 2025-02-14 12:16:59,118 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8002.73 MB 2025-02-14 12:16:59,118 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24726.78 MB 2025-02-14 12:16:59,262 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7575] 2025-02-14 12:16:59,263 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:16:59,263 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 12:16:59,264 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:16:59,264 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 12:16:59,269 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 12:16:59,270 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:16:59,270 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 12:16:59,270 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2 ('] 2025-02-14 12:17:08,738 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:17:08,738 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 12:17:08,743 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 12:17:08,747 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:17:08,747 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1142, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 12:17:08,748 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:17:08,748 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1142, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 12:17:26,423 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 12:17:26,423 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 12:17:26,423 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.67 seconds 2025-02-14 12:17:26,423 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:17:26,423 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20926.35 MB 2025-02-14 12:17:26,423 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24967.82 MB 2025-02-14 12:17:26,423 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4041.47 MB 2025-02-14 12:17:26,423 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37228.64 MB 2025-02-14 12:17:26,423 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29129.44 MB 2025-02-14 12:17:26,423 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8099.20 MB 2025-02-14 12:17:26,424 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33795.91 MB 2025-02-14 12:17:26,503 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 12:17:26,503 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 12:17:26,503 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 12:17:26,503 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:17:26,503 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24967.82 MB 2025-02-14 12:17:26,503 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21714.75 MB 2025-02-14 12:17:26,503 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3253.07 MB 2025-02-14 12:17:26,503 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29129.44 MB 2025-02-14 12:17:26,503 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44128.27 MB 2025-02-14 12:17:26,503 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 14998.83 MB 2025-02-14 12:17:26,503 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37200.51 MB 2025-02-14 12:17:28,438 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 12:17:28,438 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 12:17:28,438 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 12:17:28,438 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:17:28,438 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21714.75 MB 2025-02-14 12:17:28,438 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22245.59 MB 2025-02-14 12:17:28,438 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 12:17:28,438 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44128.27 MB 2025-02-14 12:17:28,438 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26501.71 MB 2025-02-14 12:17:28,438 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17626.56 MB 2025-02-14 12:17:28,438 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26224.93 MB 2025-02-14 12:17:28,453 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 12:17:28,453 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 12:17:28,453 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:17:28,453 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:17:28,453 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22245.59 MB 2025-02-14 12:17:28,453 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24135.13 MB 2025-02-14 12:17:28,453 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 12:17:28,453 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26501.71 MB 2025-02-14 12:17:28,453 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28389.15 MB 2025-02-14 12:17:28,453 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 12:17:28,453 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25552.56 MB 2025-02-14 12:17:28,658 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 12:17:28,658 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 12:17:28,658 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 12:17:28,658 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:17:28,658 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24135.13 MB 2025-02-14 12:17:28,658 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26376.98 MB 2025-02-14 12:17:28,658 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 12:17:28,658 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28389.15 MB 2025-02-14 12:17:28,658 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34051.46 MB 2025-02-14 12:17:28,658 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 12:17:28,658 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31921.27 MB 2025-02-14 12:17:28,659 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 12:17:28,659 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 12:17:28,659 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 12:17:28,659 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:17:28,659 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22245.59 MB 2025-02-14 12:17:28,659 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26376.98 MB 2025-02-14 12:17:28,659 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 12:17:28,659 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26501.71 MB 2025-02-14 12:17:28,659 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34051.46 MB 2025-02-14 12:17:28,659 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-14 12:17:28,659 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31921.27 MB 2025-02-14 12:17:28,820 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 12:17:28,820 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 12:17:28,821 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 12:17:28,821 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:17:28,821 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27910.53 MB 2025-02-14 12:17:28,821 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28677.53 MB 2025-02-14 12:17:28,821 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 12:17:28,821 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34051.46 MB 2025-02-14 12:17:28,821 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34468.79 MB 2025-02-14 12:17:28,821 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 12:17:28,821 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29385.32 MB 2025-02-14 12:17:28,840 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 12:17:28,840 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 12:17:28,840 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:17:28,841 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:17:28,841 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29090.42 MB 2025-02-14 12:17:28,841 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29319.18 MB 2025-02-14 12:17:28,841 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.76 MB 2025-02-14 12:17:28,841 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34468.79 MB 2025-02-14 12:17:28,841 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34468.79 MB 2025-02-14 12:17:28,841 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:17:28,841 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29547.21 MB 2025-02-14 12:17:28,842 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 12:17:28,842 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 12:17:28,842 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.09 seconds 2025-02-14 12:17:28,842 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:17:28,842 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16947.53 MB 2025-02-14 12:17:28,842 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29519.98 MB 2025-02-14 12:17:28,842 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12572.46 MB 2025-02-14 12:17:28,842 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37228.64 MB 2025-02-14 12:17:28,842 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34468.79 MB 2025-02-14 12:17:28,842 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2759.85 MB 2025-02-14 12:17:28,842 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29547.21 MB 2025-02-14 12:17:29,116 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 12:17:29,116 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 12:17:29,116 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 12:17:29,116 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:17:29,116 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29519.98 MB 2025-02-14 12:17:29,116 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21947.72 MB 2025-02-14 12:17:29,116 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7572.26 MB 2025-02-14 12:17:29,116 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34468.79 MB 2025-02-14 12:17:29,116 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34468.79 MB 2025-02-14 12:17:29,116 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:17:29,116 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32028.27 MB 2025-02-14 12:17:29,135 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8151, cut from 8153 2025-02-14 12:17:29,135 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 12:17:29,141 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 12:17:29,141 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 12:17:29,141 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:17:29,141 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:17:29,141 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21947.72 MB 2025-02-14 12:17:29,141 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30375.06 MB 2025-02-14 12:17:29,141 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8427.34 MB 2025-02-14 12:17:29,141 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34468.79 MB 2025-02-14 12:17:29,141 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42849.01 MB 2025-02-14 12:17:29,141 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8380.22 MB 2025-02-14 12:17:29,141 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30375.06 MB 2025-02-14 12:17:29,292 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7943] 2025-02-14 12:17:29,293 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:17:29,293 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 12:17:29,294 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:17:29,294 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 12:17:29,299 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 12:17:29,300 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:17:29,300 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 12:17:29,300 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 12:17:42,176 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:17:42,176 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 12:17:42,183 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 12:17:42,189 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:17:42,189 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 193, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 12:17:42,191 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:17:42,191 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 193, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 12:17:45,348 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 12:17:45,349 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 12:17:45,349 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.15 seconds 2025-02-14 12:17:45,349 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:17:45,349 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14313.56 MB 2025-02-14 12:17:45,349 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14996.51 MB 2025-02-14 12:17:45,349 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 682.95 MB 2025-02-14 12:17:45,349 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51229.23 MB 2025-02-14 12:17:45,349 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18322.82 MB 2025-02-14 12:17:45,349 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32906.41 MB 2025-02-14 12:17:45,349 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24012.17 MB 2025-02-14 12:17:45,370 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 12:17:45,371 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 12:17:45,371 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:17:45,371 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:17:45,371 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14996.51 MB 2025-02-14 12:17:45,371 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15328.09 MB 2025-02-14 12:17:45,371 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 331.57 MB 2025-02-14 12:17:45,371 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18322.82 MB 2025-02-14 12:17:45,371 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19006.49 MB 2025-02-14 12:17:45,371 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 683.67 MB 2025-02-14 12:17:45,371 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17711.66 MB 2025-02-14 12:17:46,354 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 12:17:46,355 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 12:17:46,355 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.98 seconds 2025-02-14 12:17:46,355 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:17:46,355 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15328.09 MB 2025-02-14 12:17:46,355 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15584.22 MB 2025-02-14 12:17:46,355 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 256.13 MB 2025-02-14 12:17:46,355 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19006.49 MB 2025-02-14 12:17:46,355 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19006.49 MB 2025-02-14 12:17:46,355 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:17:46,355 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19584.50 MB 2025-02-14 12:17:46,366 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 12:17:46,366 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 12:17:46,367 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:17:46,367 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:17:46,367 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15584.22 MB 2025-02-14 12:17:46,367 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16495.70 MB 2025-02-14 12:17:46,367 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 911.48 MB 2025-02-14 12:17:46,367 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19006.49 MB 2025-02-14 12:17:46,367 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19006.49 MB 2025-02-14 12:17:46,367 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:17:46,367 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17179.61 MB 2025-02-14 12:17:46,505 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 12:17:46,505 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 12:17:46,505 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 12:17:46,505 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:17:46,505 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16495.70 MB 2025-02-14 12:17:46,505 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17577.43 MB 2025-02-14 12:17:46,505 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1081.73 MB 2025-02-14 12:17:46,505 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19006.49 MB 2025-02-14 12:17:46,505 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21749.56 MB 2025-02-14 12:17:46,505 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2743.07 MB 2025-02-14 12:17:46,505 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20253.00 MB 2025-02-14 12:17:46,506 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 12:17:46,506 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 12:17:46,506 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 12:17:46,507 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:17:46,507 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15584.22 MB 2025-02-14 12:17:46,507 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17577.43 MB 2025-02-14 12:17:46,507 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1993.21 MB 2025-02-14 12:17:46,507 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19006.49 MB 2025-02-14 12:17:46,507 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21749.56 MB 2025-02-14 12:17:46,507 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2743.07 MB 2025-02-14 12:17:46,507 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20253.00 MB 2025-02-14 12:17:46,645 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 12:17:46,645 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 12:17:46,645 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 12:17:46,645 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:17:46,645 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18317.85 MB 2025-02-14 12:17:46,645 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18687.93 MB 2025-02-14 12:17:46,645 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 370.08 MB 2025-02-14 12:17:46,645 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21749.56 MB 2025-02-14 12:17:46,645 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21950.89 MB 2025-02-14 12:17:46,645 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 201.33 MB 2025-02-14 12:17:46,645 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19032.18 MB 2025-02-14 12:17:46,665 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 12:17:46,665 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 12:17:46,665 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:17:46,665 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:17:46,665 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18887.16 MB 2025-02-14 12:17:46,665 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19116.98 MB 2025-02-14 12:17:46,665 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.83 MB 2025-02-14 12:17:46,665 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21950.89 MB 2025-02-14 12:17:46,665 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21950.89 MB 2025-02-14 12:17:46,665 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:17:46,665 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19162.49 MB 2025-02-14 12:17:46,669 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 12:17:46,669 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 12:17:46,669 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.47 seconds 2025-02-14 12:17:46,669 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:17:46,669 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13641.13 MB 2025-02-14 12:17:46,669 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19318.06 MB 2025-02-14 12:17:46,669 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5676.92 MB 2025-02-14 12:17:46,669 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51229.23 MB 2025-02-14 12:17:46,669 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21950.89 MB 2025-02-14 12:17:46,669 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29278.34 MB 2025-02-14 12:17:46,669 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19318.06 MB 2025-02-14 12:17:46,963 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 12:17:46,963 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 12:17:46,963 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-14 12:17:46,963 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:17:46,963 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14654.82 MB 2025-02-14 12:17:46,963 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17668.86 MB 2025-02-14 12:17:46,963 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-14 12:17:46,963 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21950.89 MB 2025-02-14 12:17:46,963 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21950.89 MB 2025-02-14 12:17:46,963 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:17:46,963 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17970.23 MB 2025-02-14 12:17:46,984 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 12:17:46,984 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2.'] 2025-02-14 12:17:46,992 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 12:17:46,992 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 12:17:46,992 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 12:17:46,992 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:17:46,992 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17668.86 MB 2025-02-14 12:17:46,992 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26107.88 MB 2025-02-14 12:17:46,992 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 12:17:46,992 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21950.89 MB 2025-02-14 12:17:46,992 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32440.84 MB 2025-02-14 12:17:46,992 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 12:17:46,992 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26107.88 MB 2025-02-14 12:17:47,229 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 12:17:47,232 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:17:47,232 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 12:17:47,234 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:17:47,234 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 12:17:47,241 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 12:17:47,243 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:17:47,243 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 12:17:47,243 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2.'] 2025-02-14 12:18:24,028 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:18:24,028 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 12:18:24,033 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 12:18:24,037 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:18:24,037 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 223, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 12:18:24,038 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:18:24,038 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 223, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 12:18:27,512 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 12:18:27,512 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 12:18:27,512 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.47 seconds 2025-02-14 12:18:27,512 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:18:27,512 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14522.61 MB 2025-02-14 12:18:27,512 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15311.79 MB 2025-02-14 12:18:27,512 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 789.18 MB 2025-02-14 12:18:27,512 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45025.85 MB 2025-02-14 12:18:27,512 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17364.42 MB 2025-02-14 12:18:27,512 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27661.43 MB 2025-02-14 12:18:27,512 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24221.28 MB 2025-02-14 12:18:27,528 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 12:18:27,529 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 12:18:27,529 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:18:27,529 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:18:27,529 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15311.79 MB 2025-02-14 12:18:27,529 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15568.63 MB 2025-02-14 12:18:27,529 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 256.84 MB 2025-02-14 12:18:27,529 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17364.42 MB 2025-02-14 12:18:27,529 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19520.29 MB 2025-02-14 12:18:27,529 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2155.87 MB 2025-02-14 12:18:27,529 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18246.23 MB 2025-02-14 12:18:28,524 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 12:18:28,524 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 12:18:28,524 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.99 seconds 2025-02-14 12:18:28,525 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:18:28,525 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15568.63 MB 2025-02-14 12:18:28,525 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15840.68 MB 2025-02-14 12:18:28,525 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 272.06 MB 2025-02-14 12:18:28,525 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19520.29 MB 2025-02-14 12:18:28,525 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18845.01 MB 2025-02-14 12:18:28,525 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -675.28 MB 2025-02-14 12:18:28,525 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19825.04 MB 2025-02-14 12:18:28,533 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 12:18:28,533 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 12:18:28,533 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:18:28,533 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:18:28,533 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15840.68 MB 2025-02-14 12:18:28,533 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16808.83 MB 2025-02-14 12:18:28,533 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 968.15 MB 2025-02-14 12:18:28,533 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18845.01 MB 2025-02-14 12:18:28,533 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18845.01 MB 2025-02-14 12:18:28,533 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:18:28,533 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17535.27 MB 2025-02-14 12:18:28,647 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 12:18:28,647 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 12:18:28,647 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 12:18:28,647 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:18:28,647 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16808.83 MB 2025-02-14 12:18:28,647 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17958.60 MB 2025-02-14 12:18:28,647 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1149.77 MB 2025-02-14 12:18:28,647 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18845.01 MB 2025-02-14 12:18:28,647 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21994.93 MB 2025-02-14 12:18:28,647 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3149.92 MB 2025-02-14 12:18:28,647 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20803.16 MB 2025-02-14 12:18:28,647 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 12:18:28,647 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 12:18:28,647 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 12:18:28,647 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:18:28,647 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15840.68 MB 2025-02-14 12:18:28,647 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17958.60 MB 2025-02-14 12:18:28,647 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2117.92 MB 2025-02-14 12:18:28,647 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18845.01 MB 2025-02-14 12:18:28,647 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21994.93 MB 2025-02-14 12:18:28,648 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3149.92 MB 2025-02-14 12:18:28,648 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20803.16 MB 2025-02-14 12:18:28,735 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 12:18:28,735 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 12:18:28,735 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 12:18:28,735 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:18:28,735 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18744.54 MB 2025-02-14 12:18:28,735 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19137.63 MB 2025-02-14 12:18:28,735 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 393.09 MB 2025-02-14 12:18:28,735 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21994.93 MB 2025-02-14 12:18:28,735 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22206.74 MB 2025-02-14 12:18:28,735 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 211.81 MB 2025-02-14 12:18:28,735 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19503.77 MB 2025-02-14 12:18:28,747 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 12:18:28,747 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 12:18:28,747 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:18:28,747 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:18:28,747 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19349.24 MB 2025-02-14 12:18:28,747 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19566.45 MB 2025-02-14 12:18:28,747 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 217.21 MB 2025-02-14 12:18:28,747 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22206.74 MB 2025-02-14 12:18:28,747 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22206.74 MB 2025-02-14 12:18:28,747 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:18:28,747 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19602.25 MB 2025-02-14 12:18:28,748 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 12:18:28,748 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 12:18:28,748 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.71 seconds 2025-02-14 12:18:28,748 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:18:28,748 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13745.66 MB 2025-02-14 12:18:28,748 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19766.91 MB 2025-02-14 12:18:28,748 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6021.25 MB 2025-02-14 12:18:28,748 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45025.85 MB 2025-02-14 12:18:28,748 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22208.84 MB 2025-02-14 12:18:28,748 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22817.01 MB 2025-02-14 12:18:28,748 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19766.91 MB 2025-02-14 12:18:29,015 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 12:18:29,016 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 12:18:29,016 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 12:18:29,016 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:18:29,016 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14815.44 MB 2025-02-14 12:18:29,016 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17820.26 MB 2025-02-14 12:18:29,016 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3004.82 MB 2025-02-14 12:18:29,016 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22208.84 MB 2025-02-14 12:18:29,016 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22208.84 MB 2025-02-14 12:18:29,016 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:18:29,016 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18120.71 MB 2025-02-14 12:18:29,034 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8137, cut from 8139 2025-02-14 12:18:29,034 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 12:18:29,040 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 12:18:29,040 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 12:18:29,040 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:18:29,040 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:18:29,040 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17820.26 MB 2025-02-14 12:18:29,040 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26233.78 MB 2025-02-14 12:18:29,040 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8413.52 MB 2025-02-14 12:18:29,040 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22208.84 MB 2025-02-14 12:18:29,040 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32663.14 MB 2025-02-14 12:18:29,040 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10454.30 MB 2025-02-14 12:18:29,040 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26233.78 MB 2025-02-14 12:18:29,205 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7929] 2025-02-14 12:18:29,206 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:18:29,206 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 12:18:29,207 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:18:29,207 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 12:18:29,212 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 12:18:29,213 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:18:29,213 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 12:18:29,213 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 12:18:40,795 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:18:40,795 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 12:18:40,800 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 12:18:40,804 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:18:40,804 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 666, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 12:18:40,806 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:18:40,806 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 666, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 12:18:51,116 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 12:18:51,116 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 12:18:51,117 - resource_logging.py:150 - __exit__ - DEBUG - Time: 10.30 seconds 2025-02-14 12:18:51,117 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:18:51,117 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17609.50 MB 2025-02-14 12:18:51,117 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19966.70 MB 2025-02-14 12:18:51,117 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2357.20 MB 2025-02-14 12:18:51,117 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41026.58 MB 2025-02-14 12:18:51,117 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25537.02 MB 2025-02-14 12:18:51,117 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15489.56 MB 2025-02-14 12:18:51,117 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28892.81 MB 2025-02-14 12:18:51,160 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 12:18:51,160 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 12:18:51,161 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-14 12:18:51,161 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:18:51,161 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19966.70 MB 2025-02-14 12:18:51,161 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19240.18 MB 2025-02-14 12:18:51,161 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -726.52 MB 2025-02-14 12:18:51,161 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25537.02 MB 2025-02-14 12:18:51,161 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31478.25 MB 2025-02-14 12:18:51,161 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5941.23 MB 2025-02-14 12:18:51,161 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28751.22 MB 2025-02-14 12:18:53,077 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 12:18:53,077 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 12:18:53,077 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 12:18:53,077 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:18:53,077 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19240.18 MB 2025-02-14 12:18:53,077 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19771.02 MB 2025-02-14 12:18:53,077 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 12:18:53,077 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31478.25 MB 2025-02-14 12:18:53,077 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24595.40 MB 2025-02-14 12:18:53,077 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6882.85 MB 2025-02-14 12:18:53,077 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23750.35 MB 2025-02-14 12:18:53,091 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 12:18:53,091 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 12:18:53,091 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:18:53,091 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:18:53,091 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19771.02 MB 2025-02-14 12:18:53,091 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21660.55 MB 2025-02-14 12:18:53,091 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 12:18:53,091 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24595.40 MB 2025-02-14 12:18:53,091 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26482.84 MB 2025-02-14 12:18:53,091 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 12:18:53,091 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23077.98 MB 2025-02-14 12:18:53,303 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 12:18:53,303 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 12:18:53,303 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 12:18:53,303 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:18:53,303 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21660.55 MB 2025-02-14 12:18:53,303 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23902.41 MB 2025-02-14 12:18:53,303 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 12:18:53,303 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26482.84 MB 2025-02-14 12:18:53,303 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32145.15 MB 2025-02-14 12:18:53,303 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 12:18:53,303 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29446.69 MB 2025-02-14 12:18:53,304 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 12:18:53,304 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 12:18:53,304 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 12:18:53,304 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:18:53,304 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19771.02 MB 2025-02-14 12:18:53,304 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23902.41 MB 2025-02-14 12:18:53,304 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 12:18:53,304 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24595.40 MB 2025-02-14 12:18:53,304 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32145.15 MB 2025-02-14 12:18:53,304 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-14 12:18:53,304 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29446.69 MB 2025-02-14 12:18:53,474 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 12:18:53,474 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 12:18:53,474 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 12:18:53,474 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:18:53,474 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25435.95 MB 2025-02-14 12:18:53,474 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26202.95 MB 2025-02-14 12:18:53,474 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 12:18:53,474 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32145.15 MB 2025-02-14 12:18:53,474 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32562.48 MB 2025-02-14 12:18:53,474 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 12:18:53,474 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26910.74 MB 2025-02-14 12:18:53,493 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 12:18:53,493 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 12:18:53,493 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:18:53,493 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:18:53,493 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26615.84 MB 2025-02-14 12:18:53,493 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26843.93 MB 2025-02-14 12:18:53,493 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.09 MB 2025-02-14 12:18:53,493 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32562.48 MB 2025-02-14 12:18:53,493 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32562.48 MB 2025-02-14 12:18:53,493 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:18:53,493 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27077.22 MB 2025-02-14 12:18:53,495 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 12:18:53,495 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 12:18:53,495 - resource_logging.py:150 - __exit__ - DEBUG - Time: 12.69 seconds 2025-02-14 12:18:53,495 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:18:53,495 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15289.10 MB 2025-02-14 12:18:53,495 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27045.01 MB 2025-02-14 12:18:53,495 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11755.90 MB 2025-02-14 12:18:53,495 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41026.58 MB 2025-02-14 12:18:53,495 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32562.48 MB 2025-02-14 12:18:53,495 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8464.11 MB 2025-02-14 12:18:53,495 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27077.22 MB 2025-02-14 12:18:53,763 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 12:18:53,763 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 12:18:53,763 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 12:18:53,763 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:18:53,763 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27045.01 MB 2025-02-14 12:18:53,763 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20293.49 MB 2025-02-14 12:18:53,763 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6751.51 MB 2025-02-14 12:18:53,763 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32562.48 MB 2025-02-14 12:18:53,764 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32562.48 MB 2025-02-14 12:18:53,764 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:18:53,764 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29556.67 MB 2025-02-14 12:18:53,782 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 12:18:53,782 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 12:18:53,788 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 12:18:53,788 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 12:18:53,788 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:18:53,788 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:18:53,788 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20293.49 MB 2025-02-14 12:18:53,788 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28732.52 MB 2025-02-14 12:18:53,788 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 12:18:53,788 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32562.48 MB 2025-02-14 12:18:53,788 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40953.18 MB 2025-02-14 12:18:53,788 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 12:18:53,788 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28732.52 MB 2025-02-14 12:18:53,950 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 12:18:53,951 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:18:53,951 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 12:18:53,952 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:18:53,952 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 12:18:53,957 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 12:18:53,958 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:18:53,958 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 12:18:53,958 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 12:19:14,037 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:19:14,037 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 12:19:14,042 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 12:19:14,045 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:19:14,045 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 220, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 12:19:14,046 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:19:14,046 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 220, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 12:19:17,490 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 12:19:17,490 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 12:19:17,490 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.44 seconds 2025-02-14 12:19:17,490 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:19:17,490 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14501.70 MB 2025-02-14 12:19:17,490 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15280.27 MB 2025-02-14 12:19:17,490 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 778.57 MB 2025-02-14 12:19:17,490 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53538.19 MB 2025-02-14 12:19:17,490 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17379.10 MB 2025-02-14 12:19:17,490 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -36159.09 MB 2025-02-14 12:19:17,490 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24200.37 MB 2025-02-14 12:19:17,507 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 12:19:17,507 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 12:19:17,507 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:19:17,507 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:19:17,507 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15280.27 MB 2025-02-14 12:19:17,507 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15657.42 MB 2025-02-14 12:19:17,507 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 377.15 MB 2025-02-14 12:19:17,507 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17379.10 MB 2025-02-14 12:19:17,507 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19706.94 MB 2025-02-14 12:19:17,507 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2327.84 MB 2025-02-14 12:19:17,507 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18413.93 MB 2025-02-14 12:19:18,578 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 12:19:18,578 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 12:19:18,578 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.07 seconds 2025-02-14 12:19:18,578 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:19:18,578 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15657.42 MB 2025-02-14 12:19:18,578 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15949.38 MB 2025-02-14 12:19:18,578 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 291.96 MB 2025-02-14 12:19:18,578 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19706.94 MB 2025-02-14 12:19:18,578 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18561.89 MB 2025-02-14 12:19:18,578 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1145.04 MB 2025-02-14 12:19:18,578 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19913.83 MB 2025-02-14 12:19:18,587 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 12:19:18,587 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 12:19:18,587 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:19:18,587 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:19:18,587 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15949.38 MB 2025-02-14 12:19:18,587 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16990.47 MB 2025-02-14 12:19:18,587 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1041.09 MB 2025-02-14 12:19:18,587 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18561.89 MB 2025-02-14 12:19:18,587 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19602.08 MB 2025-02-14 12:19:18,587 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1040.19 MB 2025-02-14 12:19:18,587 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17770.06 MB 2025-02-14 12:19:18,711 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 12:19:18,711 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 12:19:18,711 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 12:19:18,711 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:19:18,711 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16990.47 MB 2025-02-14 12:19:18,711 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18224.57 MB 2025-02-14 12:19:18,711 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1234.10 MB 2025-02-14 12:19:18,711 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19602.08 MB 2025-02-14 12:19:18,711 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22722.64 MB 2025-02-14 12:19:18,711 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3120.56 MB 2025-02-14 12:19:18,711 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21278.09 MB 2025-02-14 12:19:18,711 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 12:19:18,712 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 12:19:18,712 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 12:19:18,712 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:19:18,712 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15949.38 MB 2025-02-14 12:19:18,712 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18224.57 MB 2025-02-14 12:19:18,712 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2275.19 MB 2025-02-14 12:19:18,712 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18561.89 MB 2025-02-14 12:19:18,712 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22722.64 MB 2025-02-14 12:19:18,712 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4160.75 MB 2025-02-14 12:19:18,712 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21278.09 MB 2025-02-14 12:19:18,805 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 12:19:18,805 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 12:19:18,805 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 12:19:18,805 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:19:18,805 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19068.02 MB 2025-02-14 12:19:18,805 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19490.92 MB 2025-02-14 12:19:18,805 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 422.90 MB 2025-02-14 12:19:18,805 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22722.64 MB 2025-02-14 12:19:18,805 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22951.23 MB 2025-02-14 12:19:18,805 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 228.59 MB 2025-02-14 12:19:18,805 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19881.55 MB 2025-02-14 12:19:18,817 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 12:19:18,817 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 12:19:18,817 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:19:18,817 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:19:18,817 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19718.01 MB 2025-02-14 12:19:18,817 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19928.89 MB 2025-02-14 12:19:18,817 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 210.88 MB 2025-02-14 12:19:18,817 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22951.23 MB 2025-02-14 12:19:18,817 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22953.33 MB 2025-02-14 12:19:18,817 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-14 12:19:18,818 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19976.62 MB 2025-02-14 12:19:18,819 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 12:19:18,819 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 12:19:18,819 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.77 seconds 2025-02-14 12:19:18,819 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:19:18,819 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13735.20 MB 2025-02-14 12:19:18,819 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20129.97 MB 2025-02-14 12:19:18,819 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6394.76 MB 2025-02-14 12:19:18,819 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53538.19 MB 2025-02-14 12:19:18,819 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22953.33 MB 2025-02-14 12:19:18,819 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30584.86 MB 2025-02-14 12:19:18,819 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20129.97 MB 2025-02-14 12:19:19,087 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 12:19:19,087 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 12:19:19,087 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 12:19:19,087 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:19:19,087 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14876.61 MB 2025-02-14 12:19:19,087 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17890.64 MB 2025-02-14 12:19:19,087 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-14 12:19:19,087 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22953.33 MB 2025-02-14 12:19:19,087 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22953.33 MB 2025-02-14 12:19:19,087 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:19:19,087 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18192.01 MB 2025-02-14 12:19:19,105 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 12:19:19,105 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2.'] 2025-02-14 12:19:19,111 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 12:19:19,111 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 12:19:19,111 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:19:19,111 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:19:19,111 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17890.64 MB 2025-02-14 12:19:19,111 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26329.67 MB 2025-02-14 12:19:19,111 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 12:19:19,111 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22953.33 MB 2025-02-14 12:19:19,111 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33443.28 MB 2025-02-14 12:19:19,111 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 12:19:19,111 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26329.67 MB 2025-02-14 12:19:19,275 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 12:19:19,277 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:19:19,277 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 12:19:19,278 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:19:19,278 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 12:19:19,282 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 12:19:19,283 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:19:19,284 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 12:19:19,284 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2.'] 2025-02-14 12:19:29,630 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:19:29,630 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 12:19:29,635 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 12:19:29,638 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:19:29,638 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 350, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 12:19:29,639 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:19:29,639 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 350, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 12:19:35,098 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 12:19:35,098 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 12:19:35,098 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.45 seconds 2025-02-14 12:19:35,098 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:19:35,098 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15407.56 MB 2025-02-14 12:19:35,098 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16646.98 MB 2025-02-14 12:19:35,098 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1239.42 MB 2025-02-14 12:19:35,098 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46028.29 MB 2025-02-14 12:19:35,098 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18926.80 MB 2025-02-14 12:19:35,098 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27101.50 MB 2025-02-14 12:19:35,098 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25559.22 MB 2025-02-14 12:19:35,122 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 12:19:35,122 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 12:19:35,122 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:19:35,122 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:19:35,122 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16646.98 MB 2025-02-14 12:19:35,122 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17119.83 MB 2025-02-14 12:19:35,122 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 472.85 MB 2025-02-14 12:19:35,122 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18926.80 MB 2025-02-14 12:19:35,122 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23282.58 MB 2025-02-14 12:19:35,122 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4355.78 MB 2025-02-14 12:19:35,122 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21359.81 MB 2025-02-14 12:19:36,742 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 12:19:36,742 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 12:19:36,742 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.62 seconds 2025-02-14 12:19:36,742 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:19:36,742 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17119.83 MB 2025-02-14 12:19:36,742 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17560.42 MB 2025-02-14 12:19:36,742 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 440.60 MB 2025-02-14 12:19:36,742 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23282.58 MB 2025-02-14 12:19:36,742 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19434.31 MB 2025-02-14 12:19:36,742 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3848.27 MB 2025-02-14 12:19:36,742 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21547.14 MB 2025-02-14 12:19:36,757 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 12:19:36,757 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 12:19:36,757 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:19:36,757 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:19:36,757 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17560.42 MB 2025-02-14 12:19:36,757 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19130.67 MB 2025-02-14 12:19:36,757 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1570.24 MB 2025-02-14 12:19:36,757 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19434.31 MB 2025-02-14 12:19:36,757 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21787.31 MB 2025-02-14 12:19:36,757 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2353.00 MB 2025-02-14 12:19:36,757 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20308.18 MB 2025-02-14 12:19:36,935 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 12:19:36,935 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 12:19:36,935 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-14 12:19:36,935 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:19:36,935 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19130.67 MB 2025-02-14 12:19:36,935 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20993.52 MB 2025-02-14 12:19:36,935 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1862.85 MB 2025-02-14 12:19:36,935 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21787.31 MB 2025-02-14 12:19:36,935 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27277.66 MB 2025-02-14 12:19:36,935 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5490.34 MB 2025-02-14 12:19:36,935 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25600.50 MB 2025-02-14 12:19:36,936 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 12:19:36,936 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 12:19:36,936 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.19 seconds 2025-02-14 12:19:36,936 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:19:36,936 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17560.42 MB 2025-02-14 12:19:36,936 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20993.52 MB 2025-02-14 12:19:36,936 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3433.09 MB 2025-02-14 12:19:36,936 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19434.31 MB 2025-02-14 12:19:36,936 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27277.66 MB 2025-02-14 12:19:36,936 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7843.35 MB 2025-02-14 12:19:36,936 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25600.50 MB 2025-02-14 12:19:37,076 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 12:19:37,076 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 12:19:37,076 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-14 12:19:37,076 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:19:37,076 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22266.36 MB 2025-02-14 12:19:37,076 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22903.49 MB 2025-02-14 12:19:37,076 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 637.14 MB 2025-02-14 12:19:37,076 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27277.66 MB 2025-02-14 12:19:37,076 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27623.69 MB 2025-02-14 12:19:37,076 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 346.03 MB 2025-02-14 12:19:37,076 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23490.96 MB 2025-02-14 12:19:37,093 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 12:19:37,093 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 12:19:37,093 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:19:37,093 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:19:37,093 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23246.19 MB 2025-02-14 12:19:37,093 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23460.74 MB 2025-02-14 12:19:37,093 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 214.55 MB 2025-02-14 12:19:37,093 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27623.69 MB 2025-02-14 12:19:37,093 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27623.69 MB 2025-02-14 12:19:37,093 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:19:37,093 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23626.61 MB 2025-02-14 12:19:37,094 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 12:19:37,094 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 12:19:37,094 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.45 seconds 2025-02-14 12:19:37,094 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:19:37,094 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14188.13 MB 2025-02-14 12:19:37,094 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23661.81 MB 2025-02-14 12:19:37,094 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 9473.68 MB 2025-02-14 12:19:37,094 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46028.29 MB 2025-02-14 12:19:37,094 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27623.69 MB 2025-02-14 12:19:37,094 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18404.61 MB 2025-02-14 12:19:37,094 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23661.81 MB 2025-02-14 12:19:37,363 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 12:19:37,363 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 12:19:37,363 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 12:19:37,363 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:19:37,363 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23661.81 MB 2025-02-14 12:19:37,363 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26675.85 MB 2025-02-14 12:19:37,363 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-14 12:19:37,363 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27623.69 MB 2025-02-14 12:19:37,363 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28160.56 MB 2025-02-14 12:19:37,363 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 536.87 MB 2025-02-14 12:19:37,363 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26977.48 MB 2025-02-14 12:19:37,382 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 12:19:37,382 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 12:19:37,388 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 12:19:37,388 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 12:19:37,388 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:19:37,388 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:19:37,388 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18872.14 MB 2025-02-14 12:19:37,388 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27311.16 MB 2025-02-14 12:19:37,388 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 12:19:37,388 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28160.56 MB 2025-02-14 12:19:37,388 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38650.51 MB 2025-02-14 12:19:37,388 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 12:19:37,388 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27311.16 MB 2025-02-14 12:19:37,553 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 12:19:37,554 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:19:37,554 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 12:19:37,555 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:19:37,555 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 12:19:37,560 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 12:19:37,561 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:19:37,561 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 12:19:37,561 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 12:21:00,490 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:21:00,490 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 12:21:00,495 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 12:21:00,499 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:21:00,499 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 179, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 12:21:00,500 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:21:00,500 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 179, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 12:21:03,259 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 12:21:03,259 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 12:21:03,259 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.75 seconds 2025-02-14 12:21:03,259 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:21:03,259 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14216.01 MB 2025-02-14 12:21:03,259 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14849.48 MB 2025-02-14 12:21:03,259 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 633.47 MB 2025-02-14 12:21:03,259 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51235.52 MB 2025-02-14 12:21:03,259 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18085.84 MB 2025-02-14 12:21:03,259 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33149.68 MB 2025-02-14 12:21:03,259 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23688.19 MB 2025-02-14 12:21:03,272 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 12:21:03,272 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 12:21:03,272 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:21:03,272 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:21:03,272 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14849.48 MB 2025-02-14 12:21:03,272 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15143.00 MB 2025-02-14 12:21:03,273 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 293.52 MB 2025-02-14 12:21:03,273 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18085.84 MB 2025-02-14 12:21:03,273 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18712.89 MB 2025-02-14 12:21:03,273 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 627.05 MB 2025-02-14 12:21:03,273 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17346.97 MB 2025-02-14 12:21:04,113 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 12:21:04,113 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 12:21:04,113 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.84 seconds 2025-02-14 12:21:04,113 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:21:04,113 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15143.00 MB 2025-02-14 12:21:04,113 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15377.90 MB 2025-02-14 12:21:04,113 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 234.90 MB 2025-02-14 12:21:04,113 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18712.89 MB 2025-02-14 12:21:04,113 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18712.89 MB 2025-02-14 12:21:04,113 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:21:04,113 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19314.48 MB 2025-02-14 12:21:04,121 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 12:21:04,121 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 12:21:04,121 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:21:04,121 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:21:04,121 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15377.84 MB 2025-02-14 12:21:04,121 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16213.75 MB 2025-02-14 12:21:04,121 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 835.92 MB 2025-02-14 12:21:04,121 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18712.89 MB 2025-02-14 12:21:04,121 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18712.89 MB 2025-02-14 12:21:04,121 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:21:04,121 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16840.97 MB 2025-02-14 12:21:04,220 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 12:21:04,220 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 12:21:04,220 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 12:21:04,220 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:21:04,220 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16213.75 MB 2025-02-14 12:21:04,220 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17205.96 MB 2025-02-14 12:21:04,220 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 992.21 MB 2025-02-14 12:21:04,220 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18712.89 MB 2025-02-14 12:21:04,220 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21019.75 MB 2025-02-14 12:21:04,220 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2306.87 MB 2025-02-14 12:21:04,220 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19660.18 MB 2025-02-14 12:21:04,221 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 12:21:04,221 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 12:21:04,221 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 12:21:04,221 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:21:04,221 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15377.84 MB 2025-02-14 12:21:04,221 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17205.96 MB 2025-02-14 12:21:04,221 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1828.12 MB 2025-02-14 12:21:04,221 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18712.89 MB 2025-02-14 12:21:04,221 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21019.75 MB 2025-02-14 12:21:04,221 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2306.87 MB 2025-02-14 12:21:04,221 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19660.18 MB 2025-02-14 12:21:04,296 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 12:21:04,296 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 12:21:04,296 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 12:21:04,296 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:21:04,296 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17884.55 MB 2025-02-14 12:21:04,296 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18224.86 MB 2025-02-14 12:21:04,296 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 340.32 MB 2025-02-14 12:21:04,296 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21019.75 MB 2025-02-14 12:21:04,296 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21200.11 MB 2025-02-14 12:21:04,296 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 180.36 MB 2025-02-14 12:21:04,296 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18543.10 MB 2025-02-14 12:21:04,306 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 12:21:04,306 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 12:21:04,306 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:21:04,306 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:21:04,306 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18407.58 MB 2025-02-14 12:21:04,306 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18635.86 MB 2025-02-14 12:21:04,306 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.28 MB 2025-02-14 12:21:04,306 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21200.11 MB 2025-02-14 12:21:04,306 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21200.11 MB 2025-02-14 12:21:04,306 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:21:04,306 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18653.75 MB 2025-02-14 12:21:04,307 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 12:21:04,307 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 12:21:04,307 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.80 seconds 2025-02-14 12:21:04,307 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:21:04,307 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13592.36 MB 2025-02-14 12:21:04,307 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18836.76 MB 2025-02-14 12:21:04,307 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5244.40 MB 2025-02-14 12:21:04,307 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51235.52 MB 2025-02-14 12:21:04,307 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21200.11 MB 2025-02-14 12:21:04,307 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30035.41 MB 2025-02-14 12:21:04,307 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18836.76 MB 2025-02-14 12:21:04,574 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 12:21:04,574 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 12:21:04,574 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 12:21:04,574 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:21:04,574 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18836.76 MB 2025-02-14 12:21:04,574 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17542.94 MB 2025-02-14 12:21:04,574 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1293.82 MB 2025-02-14 12:21:04,574 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21200.11 MB 2025-02-14 12:21:04,574 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21200.11 MB 2025-02-14 12:21:04,574 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:21:04,574 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19070.98 MB 2025-02-14 12:21:04,592 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8155, cut from 8157 2025-02-14 12:21:04,592 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 12:21:04,598 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 12:21:04,598 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 12:21:04,598 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:21:04,598 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:21:04,598 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17542.94 MB 2025-02-14 12:21:04,598 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25974.40 MB 2025-02-14 12:21:04,598 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8431.46 MB 2025-02-14 12:21:04,598 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21200.11 MB 2025-02-14 12:21:04,598 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31681.68 MB 2025-02-14 12:21:04,598 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10481.57 MB 2025-02-14 12:21:04,598 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25974.40 MB 2025-02-14 12:21:04,749 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7947] 2025-02-14 12:21:04,750 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:21:04,750 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 12:21:04,751 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:21:04,751 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 12:21:04,755 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 12:21:04,756 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:21:04,756 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 12:21:04,757 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 12:21:27,390 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:21:27,390 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 12:21:27,395 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 12:21:27,399 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:21:27,399 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1780, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 12:21:27,400 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:21:27,400 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1780, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 12:21:54,888 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 12:21:54,889 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 12:21:54,889 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.48 seconds 2025-02-14 12:21:54,889 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:21:54,889 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25372.03 MB 2025-02-14 12:21:54,889 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31671.88 MB 2025-02-14 12:21:54,889 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6299.84 MB 2025-02-14 12:21:54,889 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40066.09 MB 2025-02-14 12:21:54,889 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39978.01 MB 2025-02-14 12:21:54,889 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -88.08 MB 2025-02-14 12:21:54,889 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40506.52 MB 2025-02-14 12:21:55,000 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 12:21:55,000 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 12:21:55,000 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 12:21:55,000 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:21:55,000 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31671.88 MB 2025-02-14 12:21:55,000 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25031.52 MB 2025-02-14 12:21:55,000 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6640.36 MB 2025-02-14 12:21:55,000 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39978.01 MB 2025-02-14 12:21:55,000 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58223.23 MB 2025-02-14 12:21:55,000 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 18245.22 MB 2025-02-14 12:21:55,000 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49315.92 MB 2025-02-14 12:21:56,965 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 12:21:56,966 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 12:21:56,966 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.96 seconds 2025-02-14 12:21:56,966 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:21:56,966 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25031.52 MB 2025-02-14 12:21:56,966 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25562.36 MB 2025-02-14 12:21:56,966 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 12:21:56,966 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58223.23 MB 2025-02-14 12:21:56,966 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35093.74 MB 2025-02-14 12:21:56,966 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23129.49 MB 2025-02-14 12:21:56,966 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29541.69 MB 2025-02-14 12:21:56,980 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 12:21:56,980 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 12:21:56,980 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:21:56,980 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:21:56,980 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25562.36 MB 2025-02-14 12:21:56,980 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27451.89 MB 2025-02-14 12:21:56,980 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 12:21:56,980 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35093.74 MB 2025-02-14 12:21:56,980 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35093.74 MB 2025-02-14 12:21:56,980 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:21:56,980 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28869.32 MB 2025-02-14 12:21:57,199 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 12:21:57,199 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 12:21:57,199 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 12:21:57,199 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:21:57,199 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27451.89 MB 2025-02-14 12:21:57,199 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29693.75 MB 2025-02-14 12:21:57,199 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 12:21:57,199 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35093.74 MB 2025-02-14 12:21:57,199 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38396.76 MB 2025-02-14 12:21:57,199 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 12:21:57,199 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35238.03 MB 2025-02-14 12:21:57,200 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 12:21:57,200 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 12:21:57,200 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 12:21:57,200 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:21:57,200 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25562.36 MB 2025-02-14 12:21:57,200 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29693.75 MB 2025-02-14 12:21:57,200 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 12:21:57,200 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35093.74 MB 2025-02-14 12:21:57,200 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38396.76 MB 2025-02-14 12:21:57,200 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 12:21:57,200 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35238.03 MB 2025-02-14 12:21:57,372 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 12:21:57,372 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 12:21:57,372 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 12:21:57,372 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:21:57,372 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31227.29 MB 2025-02-14 12:21:57,372 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31994.29 MB 2025-02-14 12:21:57,372 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 12:21:57,372 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38396.76 MB 2025-02-14 12:21:57,372 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38811.99 MB 2025-02-14 12:21:57,372 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 12:21:57,372 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32702.08 MB 2025-02-14 12:21:57,393 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 12:21:57,393 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 12:21:57,393 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:21:57,393 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:21:57,393 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32407.18 MB 2025-02-14 12:21:57,393 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32635.53 MB 2025-02-14 12:21:57,393 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.35 MB 2025-02-14 12:21:57,393 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38811.99 MB 2025-02-14 12:21:57,393 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38811.99 MB 2025-02-14 12:21:57,393 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:21:57,393 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32851.22 MB 2025-02-14 12:21:57,394 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 12:21:57,394 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 12:21:57,394 - resource_logging.py:150 - __exit__ - DEBUG - Time: 29.99 seconds 2025-02-14 12:21:57,394 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:21:57,394 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19170.37 MB 2025-02-14 12:21:57,394 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32835.79 MB 2025-02-14 12:21:57,394 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13665.42 MB 2025-02-14 12:21:57,394 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40066.09 MB 2025-02-14 12:21:57,394 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38811.99 MB 2025-02-14 12:21:57,394 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1254.10 MB 2025-02-14 12:21:57,394 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32851.22 MB 2025-02-14 12:21:57,664 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 12:21:57,664 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 12:21:57,664 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 12:21:57,664 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:21:57,664 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32835.79 MB 2025-02-14 12:21:57,664 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24162.19 MB 2025-02-14 12:21:57,664 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8673.60 MB 2025-02-14 12:21:57,664 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38811.99 MB 2025-02-14 12:21:57,664 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38811.99 MB 2025-02-14 12:21:57,664 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:21:57,664 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35337.32 MB 2025-02-14 12:21:57,682 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8129, cut from 8131 2025-02-14 12:21:57,682 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 12:21:57,688 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 12:21:57,688 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 12:21:57,688 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:21:57,688 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:21:57,688 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24162.19 MB 2025-02-14 12:21:57,688 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32567.30 MB 2025-02-14 12:21:57,688 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8405.11 MB 2025-02-14 12:21:57,688 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38811.99 MB 2025-02-14 12:21:57,689 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47169.14 MB 2025-02-14 12:21:57,689 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8357.15 MB 2025-02-14 12:21:57,689 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32567.30 MB 2025-02-14 12:21:57,851 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7921] 2025-02-14 12:21:57,852 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:21:57,853 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 12:21:57,853 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:21:57,854 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 12:21:57,858 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 12:21:57,859 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:21:57,859 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 12:21:57,859 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 12:22:36,939 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:22:36,939 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 12:22:36,944 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 12:22:36,948 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:22:36,948 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 534, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 12:22:36,949 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:22:36,949 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 534, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 12:22:45,229 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 12:22:45,229 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 12:22:45,229 - resource_logging.py:150 - __exit__ - DEBUG - Time: 8.28 seconds 2025-02-14 12:22:45,230 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:22:45,230 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16689.70 MB 2025-02-14 12:22:45,230 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18579.50 MB 2025-02-14 12:22:45,230 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.80 MB 2025-02-14 12:22:45,230 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59703.82 MB 2025-02-14 12:22:45,230 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22991.08 MB 2025-02-14 12:22:45,230 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -36712.74 MB 2025-02-14 12:22:45,230 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27520.03 MB 2025-02-14 12:22:45,267 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 12:22:45,267 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 12:22:45,267 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-14 12:22:45,267 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:22:45,267 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18579.50 MB 2025-02-14 12:22:45,267 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18553.95 MB 2025-02-14 12:22:45,267 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -25.55 MB 2025-02-14 12:22:45,267 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22991.08 MB 2025-02-14 12:22:45,267 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29330.77 MB 2025-02-14 12:22:45,267 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6339.69 MB 2025-02-14 12:22:45,267 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26330.06 MB 2025-02-14 12:22:47,183 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 12:22:47,183 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 12:22:47,183 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 12:22:47,183 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:22:47,183 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18553.95 MB 2025-02-14 12:22:47,183 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19084.79 MB 2025-02-14 12:22:47,183 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 12:22:47,183 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29330.77 MB 2025-02-14 12:22:47,183 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24406.65 MB 2025-02-14 12:22:47,183 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4924.11 MB 2025-02-14 12:22:47,183 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23064.13 MB 2025-02-14 12:22:47,197 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 12:22:47,197 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 12:22:47,197 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:22:47,197 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:22:47,197 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19084.79 MB 2025-02-14 12:22:47,197 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20974.33 MB 2025-02-14 12:22:47,197 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 12:22:47,197 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24406.65 MB 2025-02-14 12:22:47,197 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25350.37 MB 2025-02-14 12:22:47,197 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 12:22:47,197 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22391.75 MB 2025-02-14 12:22:47,411 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 12:22:47,411 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 12:22:47,411 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 12:22:47,411 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:22:47,411 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20974.33 MB 2025-02-14 12:22:47,411 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23216.18 MB 2025-02-14 12:22:47,411 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 12:22:47,411 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25350.37 MB 2025-02-14 12:22:47,411 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31012.68 MB 2025-02-14 12:22:47,411 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 12:22:47,411 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28760.46 MB 2025-02-14 12:22:47,412 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 12:22:47,412 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 12:22:47,412 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 12:22:47,412 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:22:47,412 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19084.79 MB 2025-02-14 12:22:47,412 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23216.18 MB 2025-02-14 12:22:47,412 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 12:22:47,412 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24406.65 MB 2025-02-14 12:22:47,412 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31012.68 MB 2025-02-14 12:22:47,412 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 12:22:47,412 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28760.46 MB 2025-02-14 12:22:47,584 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 12:22:47,584 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 12:22:47,584 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 12:22:47,584 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:22:47,584 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24749.72 MB 2025-02-14 12:22:47,584 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25516.73 MB 2025-02-14 12:22:47,584 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 12:22:47,584 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31012.68 MB 2025-02-14 12:22:47,584 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31425.82 MB 2025-02-14 12:22:47,584 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 12:22:47,584 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26224.51 MB 2025-02-14 12:22:47,603 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 12:22:47,603 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 12:22:47,603 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:22:47,604 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:22:47,604 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25929.61 MB 2025-02-14 12:22:47,604 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26158.02 MB 2025-02-14 12:22:47,604 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.40 MB 2025-02-14 12:22:47,604 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31425.82 MB 2025-02-14 12:22:47,604 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31425.82 MB 2025-02-14 12:22:47,604 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:22:47,604 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26354.06 MB 2025-02-14 12:22:47,605 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 12:22:47,605 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 12:22:47,605 - resource_logging.py:150 - __exit__ - DEBUG - Time: 10.65 seconds 2025-02-14 12:22:47,605 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:22:47,605 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14829.20 MB 2025-02-14 12:22:47,605 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26358.85 MB 2025-02-14 12:22:47,605 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11529.64 MB 2025-02-14 12:22:47,605 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59703.82 MB 2025-02-14 12:22:47,605 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31425.82 MB 2025-02-14 12:22:47,605 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28278.00 MB 2025-02-14 12:22:47,605 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26358.85 MB 2025-02-14 12:22:47,874 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 12:22:47,874 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 12:22:47,874 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 12:22:47,874 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:22:47,874 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26358.85 MB 2025-02-14 12:22:47,874 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19829.79 MB 2025-02-14 12:22:47,874 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6529.06 MB 2025-02-14 12:22:47,874 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31425.82 MB 2025-02-14 12:22:47,874 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31425.82 MB 2025-02-14 12:22:47,874 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:22:47,874 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28867.44 MB 2025-02-14 12:22:47,891 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8152, cut from 8154 2025-02-14 12:22:47,892 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 12:22:47,898 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 12:22:47,898 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 12:22:47,898 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:22:47,898 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:22:47,898 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19829.79 MB 2025-02-14 12:22:47,898 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28258.91 MB 2025-02-14 12:22:47,898 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8429.12 MB 2025-02-14 12:22:47,898 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31425.82 MB 2025-02-14 12:22:47,898 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39806.04 MB 2025-02-14 12:22:47,898 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8380.22 MB 2025-02-14 12:22:47,898 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28258.91 MB 2025-02-14 12:22:48,062 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7944] 2025-02-14 12:22:48,063 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:22:48,064 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 12:22:48,064 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:22:48,064 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 12:22:48,069 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 12:22:48,070 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:22:48,070 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 12:22:48,070 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 12:24:04,076 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:24:04,076 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 12:24:04,081 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 12:24:04,085 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:24:04,085 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 876, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 12:24:04,086 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:24:04,086 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 876, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 12:24:17,518 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 12:24:17,518 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 12:24:17,518 - resource_logging.py:150 - __exit__ - DEBUG - Time: 13.43 seconds 2025-02-14 12:24:17,518 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:24:17,519 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19072.82 MB 2025-02-14 12:24:17,519 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22172.93 MB 2025-02-14 12:24:17,519 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3100.11 MB 2025-02-14 12:24:17,519 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48186.26 MB 2025-02-14 12:24:17,519 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28393.34 MB 2025-02-14 12:24:17,519 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19792.92 MB 2025-02-14 12:24:17,519 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31036.41 MB 2025-02-14 12:24:17,588 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 12:24:17,588 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 12:24:17,588 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 12:24:17,588 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:24:17,588 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22172.93 MB 2025-02-14 12:24:17,588 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20331.90 MB 2025-02-14 12:24:17,589 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1841.03 MB 2025-02-14 12:24:17,589 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28393.34 MB 2025-02-14 12:24:17,589 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36677.09 MB 2025-02-14 12:24:17,589 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8283.75 MB 2025-02-14 12:24:17,589 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31891.62 MB 2025-02-14 12:24:19,497 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 12:24:19,497 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 12:24:19,497 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 12:24:19,497 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:24:19,497 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20331.90 MB 2025-02-14 12:24:19,497 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20862.74 MB 2025-02-14 12:24:19,497 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 12:24:19,497 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36677.09 MB 2025-02-14 12:24:19,497 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26707.23 MB 2025-02-14 12:24:19,497 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9969.86 MB 2025-02-14 12:24:19,497 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24842.08 MB 2025-02-14 12:24:19,510 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 12:24:19,510 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 12:24:19,510 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:24:19,510 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:24:19,510 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20862.74 MB 2025-02-14 12:24:19,510 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22752.28 MB 2025-02-14 12:24:19,510 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 12:24:19,510 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26707.23 MB 2025-02-14 12:24:19,510 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26707.23 MB 2025-02-14 12:24:19,510 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:24:19,510 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24169.71 MB 2025-02-14 12:24:19,719 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 12:24:19,719 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 12:24:19,719 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 12:24:19,719 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:24:19,719 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22752.28 MB 2025-02-14 12:24:19,719 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24994.13 MB 2025-02-14 12:24:19,719 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 12:24:19,719 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26707.23 MB 2025-02-14 12:24:19,719 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33313.26 MB 2025-02-14 12:24:19,719 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 12:24:19,719 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30538.42 MB 2025-02-14 12:24:19,720 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 12:24:19,720 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 12:24:19,720 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 12:24:19,720 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:24:19,720 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20862.74 MB 2025-02-14 12:24:19,720 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24994.13 MB 2025-02-14 12:24:19,720 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 12:24:19,721 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26707.23 MB 2025-02-14 12:24:19,721 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33313.26 MB 2025-02-14 12:24:19,721 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 12:24:19,721 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30538.42 MB 2025-02-14 12:24:19,890 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 12:24:19,890 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 12:24:19,890 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 12:24:19,890 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:24:19,890 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26527.68 MB 2025-02-14 12:24:19,890 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27294.68 MB 2025-02-14 12:24:19,890 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 12:24:19,890 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33313.26 MB 2025-02-14 12:24:19,890 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33728.50 MB 2025-02-14 12:24:19,890 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 12:24:19,890 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28002.47 MB 2025-02-14 12:24:19,909 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 12:24:19,909 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 12:24:19,909 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:24:19,909 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:24:19,909 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27707.57 MB 2025-02-14 12:24:19,909 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27935.80 MB 2025-02-14 12:24:19,909 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.23 MB 2025-02-14 12:24:19,909 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33728.50 MB 2025-02-14 12:24:19,909 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33728.50 MB 2025-02-14 12:24:19,909 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:24:19,909 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28129.63 MB 2025-02-14 12:24:19,911 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 12:24:19,911 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 12:24:19,911 - resource_logging.py:150 - __exit__ - DEBUG - Time: 15.82 seconds 2025-02-14 12:24:19,911 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:24:19,911 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16020.76 MB 2025-02-14 12:24:19,911 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28135.94 MB 2025-02-14 12:24:19,911 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12115.18 MB 2025-02-14 12:24:19,911 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48186.26 MB 2025-02-14 12:24:19,911 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33728.50 MB 2025-02-14 12:24:19,911 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14457.77 MB 2025-02-14 12:24:19,911 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28135.94 MB 2025-02-14 12:24:20,178 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 12:24:20,178 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 12:24:20,178 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 12:24:20,178 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:24:20,178 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28135.94 MB 2025-02-14 12:24:20,178 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21010.67 MB 2025-02-14 12:24:20,178 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7125.26 MB 2025-02-14 12:24:20,178 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33728.50 MB 2025-02-14 12:24:20,178 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33728.50 MB 2025-02-14 12:24:20,178 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:24:20,178 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30635.93 MB 2025-02-14 12:24:20,196 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8124, cut from 8126 2025-02-14 12:24:20,196 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 12:24:20,202 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 12:24:20,202 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 12:24:20,202 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:24:20,202 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:24:20,202 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21010.67 MB 2025-02-14 12:24:20,202 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29411.54 MB 2025-02-14 12:24:20,202 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8400.86 MB 2025-02-14 12:24:20,202 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33728.50 MB 2025-02-14 12:24:20,202 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42079.35 MB 2025-02-14 12:24:20,202 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8350.86 MB 2025-02-14 12:24:20,202 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29411.54 MB 2025-02-14 12:24:20,353 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7916] 2025-02-14 12:24:20,355 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:24:20,355 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 12:24:20,356 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:24:20,356 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 12:24:20,360 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 12:24:20,361 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:24:20,361 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 12:24:20,362 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 12:25:36,115 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:25:36,115 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 12:25:36,120 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 12:25:36,124 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:25:36,124 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1668, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 12:25:36,125 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:25:36,125 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1668, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 12:26:01,859 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 12:26:01,859 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 12:26:01,859 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.73 seconds 2025-02-14 12:26:01,859 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:26:01,860 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24591.60 MB 2025-02-14 12:26:01,860 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30495.08 MB 2025-02-14 12:26:01,860 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5903.48 MB 2025-02-14 12:26:01,860 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50430.21 MB 2025-02-14 12:26:01,860 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39527.12 MB 2025-02-14 12:26:01,860 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10903.09 MB 2025-02-14 12:26:01,860 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39499.59 MB 2025-02-14 12:26:01,959 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 12:26:01,959 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 12:26:01,959 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 12:26:01,959 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:26:01,959 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30495.08 MB 2025-02-14 12:26:01,959 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24449.26 MB 2025-02-14 12:26:01,959 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6045.82 MB 2025-02-14 12:26:01,959 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39527.12 MB 2025-02-14 12:26:01,959 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56398.71 MB 2025-02-14 12:26:01,959 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 16871.59 MB 2025-02-14 12:26:01,960 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47673.91 MB 2025-02-14 12:26:03,885 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 12:26:03,885 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 12:26:03,885 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 12:26:03,885 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:26:03,885 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24449.26 MB 2025-02-14 12:26:03,885 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24980.10 MB 2025-02-14 12:26:03,885 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 12:26:03,885 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56398.71 MB 2025-02-14 12:26:03,885 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30863.79 MB 2025-02-14 12:26:03,885 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25534.92 MB 2025-02-14 12:26:03,885 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28959.44 MB 2025-02-14 12:26:03,901 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 12:26:03,901 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 12:26:03,901 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:26:03,901 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:26:03,901 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24980.10 MB 2025-02-14 12:26:03,901 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26869.64 MB 2025-02-14 12:26:03,901 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 12:26:03,901 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30863.79 MB 2025-02-14 12:26:03,901 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31807.50 MB 2025-02-14 12:26:03,901 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 12:26:03,901 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28287.07 MB 2025-02-14 12:26:04,114 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 12:26:04,114 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 12:26:04,114 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 12:26:04,114 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:26:04,114 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26869.64 MB 2025-02-14 12:26:04,114 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29111.49 MB 2025-02-14 12:26:04,114 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 12:26:04,114 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31807.50 MB 2025-02-14 12:26:04,114 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37941.67 MB 2025-02-14 12:26:04,114 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 12:26:04,114 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34655.78 MB 2025-02-14 12:26:04,115 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 12:26:04,115 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 12:26:04,115 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 12:26:04,115 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:26:04,115 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24980.10 MB 2025-02-14 12:26:04,115 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29111.49 MB 2025-02-14 12:26:04,115 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 12:26:04,115 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30863.79 MB 2025-02-14 12:26:04,115 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37941.67 MB 2025-02-14 12:26:04,115 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7077.89 MB 2025-02-14 12:26:04,115 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34655.78 MB 2025-02-14 12:26:04,286 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 12:26:04,286 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 12:26:04,286 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 12:26:04,286 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:26:04,286 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30645.04 MB 2025-02-14 12:26:04,286 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31412.04 MB 2025-02-14 12:26:04,286 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 12:26:04,286 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37941.67 MB 2025-02-14 12:26:04,286 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38359.01 MB 2025-02-14 12:26:04,286 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 12:26:04,286 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32119.83 MB 2025-02-14 12:26:04,307 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 12:26:04,307 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 12:26:04,307 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:26:04,307 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:26:04,307 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31824.93 MB 2025-02-14 12:26:04,307 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32053.68 MB 2025-02-14 12:26:04,307 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.75 MB 2025-02-14 12:26:04,307 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38359.01 MB 2025-02-14 12:26:04,307 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38359.01 MB 2025-02-14 12:26:04,307 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:26:04,307 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32268.41 MB 2025-02-14 12:26:04,308 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 12:26:04,308 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 12:26:04,308 - resource_logging.py:150 - __exit__ - DEBUG - Time: 28.18 seconds 2025-02-14 12:26:04,308 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:26:04,308 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18780.15 MB 2025-02-14 12:26:04,308 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32254.51 MB 2025-02-14 12:26:04,308 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13474.35 MB 2025-02-14 12:26:04,309 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50430.21 MB 2025-02-14 12:26:04,309 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38359.01 MB 2025-02-14 12:26:04,309 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12071.21 MB 2025-02-14 12:26:04,309 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32268.41 MB 2025-02-14 12:26:04,578 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 12:26:04,578 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 12:26:04,578 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 12:26:04,578 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:26:04,578 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32254.51 MB 2025-02-14 12:26:04,578 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23781.22 MB 2025-02-14 12:26:04,578 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8473.28 MB 2025-02-14 12:26:04,578 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38359.01 MB 2025-02-14 12:26:04,578 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38359.01 MB 2025-02-14 12:26:04,578 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:26:04,578 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34763.59 MB 2025-02-14 12:26:04,595 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8152, cut from 8154 2025-02-14 12:26:04,596 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 12:26:04,602 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 12:26:04,602 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 12:26:04,602 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:26:04,602 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:26:04,602 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23781.22 MB 2025-02-14 12:26:04,602 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32210.35 MB 2025-02-14 12:26:04,602 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8429.12 MB 2025-02-14 12:26:04,602 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38359.01 MB 2025-02-14 12:26:04,602 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46739.23 MB 2025-02-14 12:26:04,602 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8380.22 MB 2025-02-14 12:26:04,602 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32210.35 MB 2025-02-14 12:26:04,772 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7944] 2025-02-14 12:26:04,773 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:26:04,773 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 12:26:04,774 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:26:04,774 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 12:26:04,779 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 12:26:04,780 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:26:04,780 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 12:26:04,781 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 12:26:13,512 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:26:13,512 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 12:26:13,517 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 12:26:13,520 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:26:13,520 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1757, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 12:26:13,522 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:26:13,522 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1757, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 12:26:40,939 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 12:26:40,939 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 12:26:40,939 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.41 seconds 2025-02-14 12:26:40,939 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:26:40,939 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25211.76 MB 2025-02-14 12:26:40,939 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31429.82 MB 2025-02-14 12:26:40,939 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6218.06 MB 2025-02-14 12:26:40,939 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55119.45 MB 2025-02-14 12:26:40,939 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39856.37 MB 2025-02-14 12:26:40,939 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15263.07 MB 2025-02-14 12:26:40,939 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40346.25 MB 2025-02-14 12:26:41,041 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 12:26:41,041 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 12:26:41,041 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 12:26:41,041 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:26:41,041 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31429.82 MB 2025-02-14 12:26:41,041 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24911.95 MB 2025-02-14 12:26:41,041 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6517.87 MB 2025-02-14 12:26:41,041 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39856.37 MB 2025-02-14 12:26:41,041 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56417.58 MB 2025-02-14 12:26:41,041 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 16561.21 MB 2025-02-14 12:26:41,041 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47945.18 MB 2025-02-14 12:26:42,983 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 12:26:42,983 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 12:26:42,984 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-14 12:26:42,984 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:26:42,984 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24911.95 MB 2025-02-14 12:26:42,984 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25442.79 MB 2025-02-14 12:26:42,984 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 12:26:42,984 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56417.58 MB 2025-02-14 12:26:42,984 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35053.90 MB 2025-02-14 12:26:42,984 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21363.69 MB 2025-02-14 12:26:42,984 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29422.12 MB 2025-02-14 12:26:42,997 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 12:26:42,997 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 12:26:42,997 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:26:42,997 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:26:42,997 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25442.79 MB 2025-02-14 12:26:42,997 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27332.32 MB 2025-02-14 12:26:42,997 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 12:26:42,997 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35053.90 MB 2025-02-14 12:26:42,997 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35053.90 MB 2025-02-14 12:26:42,997 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:26:42,997 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28749.75 MB 2025-02-14 12:26:43,211 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 12:26:43,211 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 12:26:43,211 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 12:26:43,211 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:26:43,211 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27332.32 MB 2025-02-14 12:26:43,211 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29574.18 MB 2025-02-14 12:26:43,211 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 12:26:43,211 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35053.90 MB 2025-02-14 12:26:43,211 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38356.91 MB 2025-02-14 12:26:43,211 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 12:26:43,211 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35118.46 MB 2025-02-14 12:26:43,211 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 12:26:43,211 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 12:26:43,212 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 12:26:43,212 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:26:43,212 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25442.79 MB 2025-02-14 12:26:43,212 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29574.18 MB 2025-02-14 12:26:43,212 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 12:26:43,212 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35053.90 MB 2025-02-14 12:26:43,212 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38356.91 MB 2025-02-14 12:26:43,212 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 12:26:43,212 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35118.46 MB 2025-02-14 12:26:43,382 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 12:26:43,382 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 12:26:43,382 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 12:26:43,382 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:26:43,382 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31107.72 MB 2025-02-14 12:26:43,382 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31874.72 MB 2025-02-14 12:26:43,382 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 12:26:43,382 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38356.91 MB 2025-02-14 12:26:43,382 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38774.24 MB 2025-02-14 12:26:43,382 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 12:26:43,382 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32582.51 MB 2025-02-14 12:26:43,401 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 12:26:43,401 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 12:26:43,401 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:26:43,401 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:26:43,401 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32287.61 MB 2025-02-14 12:26:43,401 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32515.76 MB 2025-02-14 12:26:43,401 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.15 MB 2025-02-14 12:26:43,401 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38774.24 MB 2025-02-14 12:26:43,401 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38774.24 MB 2025-02-14 12:26:43,401 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:26:43,401 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32731.46 MB 2025-02-14 12:26:43,403 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 12:26:43,403 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 12:26:43,403 - resource_logging.py:150 - __exit__ - DEBUG - Time: 29.88 seconds 2025-02-14 12:26:43,403 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:26:43,403 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19090.23 MB 2025-02-14 12:26:43,403 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32715.83 MB 2025-02-14 12:26:43,403 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13625.59 MB 2025-02-14 12:26:43,403 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55119.45 MB 2025-02-14 12:26:43,403 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38774.24 MB 2025-02-14 12:26:43,403 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16345.20 MB 2025-02-14 12:26:43,403 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32731.46 MB 2025-02-14 12:26:43,674 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 12:26:43,674 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 12:26:43,674 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 12:26:43,674 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:26:43,674 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32715.83 MB 2025-02-14 12:26:43,674 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24079.01 MB 2025-02-14 12:26:43,674 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8636.82 MB 2025-02-14 12:26:43,674 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38774.24 MB 2025-02-14 12:26:43,674 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38774.24 MB 2025-02-14 12:26:43,674 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:26:43,674 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35214.90 MB 2025-02-14 12:26:43,692 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8121, cut from 8123 2025-02-14 12:26:43,692 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 12:26:43,698 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 12:26:43,699 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 12:26:43,699 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:26:43,699 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:26:43,699 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24079.01 MB 2025-02-14 12:26:43,699 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32475.77 MB 2025-02-14 12:26:43,699 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8396.77 MB 2025-02-14 12:26:43,699 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38774.24 MB 2025-02-14 12:26:43,699 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42947.58 MB 2025-02-14 12:26:43,699 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4173.33 MB 2025-02-14 12:26:43,699 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32475.77 MB 2025-02-14 12:26:43,861 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7913] 2025-02-14 12:26:43,863 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:26:43,863 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 12:26:43,864 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:26:43,864 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 12:26:43,868 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 12:26:43,869 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:26:43,869 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 12:26:43,870 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 12:26:56,913 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:26:56,913 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 12:26:56,920 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 12:26:56,926 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:26:56,927 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 136, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 12:26:56,928 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:26:56,929 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 136, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 12:26:59,163 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 12:26:59,163 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 12:26:59,163 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.23 seconds 2025-02-14 12:26:59,163 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:26:59,163 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13916.38 MB 2025-02-14 12:26:59,163 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14397.67 MB 2025-02-14 12:26:59,163 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 481.30 MB 2025-02-14 12:26:59,163 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55469.67 MB 2025-02-14 12:26:59,163 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17379.10 MB 2025-02-14 12:26:59,163 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -38090.57 MB 2025-02-14 12:26:59,163 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23388.55 MB 2025-02-14 12:26:59,177 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 12:26:59,177 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 12:26:59,177 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:26:59,177 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:26:59,177 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14397.67 MB 2025-02-14 12:26:59,177 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14588.72 MB 2025-02-14 12:26:59,177 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 191.05 MB 2025-02-14 12:26:59,177 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17379.10 MB 2025-02-14 12:26:59,177 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17379.10 MB 2025-02-14 12:26:59,177 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:26:59,177 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16252.04 MB 2025-02-14 12:26:59,829 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 12:26:59,829 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 12:26:59,829 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.65 seconds 2025-02-14 12:26:59,829 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:26:59,829 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14588.72 MB 2025-02-14 12:26:59,829 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14761.24 MB 2025-02-14 12:26:59,829 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 172.52 MB 2025-02-14 12:26:59,829 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17379.10 MB 2025-02-14 12:26:59,830 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17781.75 MB 2025-02-14 12:26:59,830 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 402.65 MB 2025-02-14 12:26:59,830 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18760.20 MB 2025-02-14 12:26:59,836 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 12:26:59,836 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 12:26:59,836 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 12:26:59,836 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:26:59,836 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14761.18 MB 2025-02-14 12:26:59,836 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15375.13 MB 2025-02-14 12:26:59,836 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 613.95 MB 2025-02-14 12:26:59,836 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17781.75 MB 2025-02-14 12:26:59,836 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17781.75 MB 2025-02-14 12:26:59,836 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:26:59,836 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15835.80 MB 2025-02-14 12:26:59,922 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 12:26:59,922 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 12:26:59,922 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 12:26:59,922 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:26:59,922 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15375.13 MB 2025-02-14 12:26:59,922 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16103.78 MB 2025-02-14 12:26:59,922 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 728.65 MB 2025-02-14 12:26:59,922 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17781.75 MB 2025-02-14 12:26:59,922 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19014.88 MB 2025-02-14 12:26:59,922 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1233.13 MB 2025-02-14 12:26:59,922 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17905.62 MB 2025-02-14 12:26:59,923 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 12:26:59,923 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 12:26:59,923 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 12:26:59,923 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:26:59,923 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14761.18 MB 2025-02-14 12:26:59,923 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16103.78 MB 2025-02-14 12:26:59,923 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1342.60 MB 2025-02-14 12:26:59,923 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17781.75 MB 2025-02-14 12:26:59,923 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19014.88 MB 2025-02-14 12:26:59,923 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1233.13 MB 2025-02-14 12:26:59,923 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17905.62 MB 2025-02-14 12:26:59,977 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 12:26:59,977 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 12:26:59,977 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-14 12:26:59,977 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:26:59,977 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16602.18 MB 2025-02-14 12:26:59,977 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16851.45 MB 2025-02-14 12:26:59,977 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 249.28 MB 2025-02-14 12:26:59,977 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19014.88 MB 2025-02-14 12:26:59,977 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19149.09 MB 2025-02-14 12:26:59,977 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 134.22 MB 2025-02-14 12:26:59,977 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17094.59 MB 2025-02-14 12:26:59,985 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 12:26:59,985 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 12:26:59,985 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:26:59,985 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:26:59,985 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16985.65 MB 2025-02-14 12:26:59,985 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17190.98 MB 2025-02-14 12:26:59,985 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.33 MB 2025-02-14 12:26:59,985 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19149.09 MB 2025-02-14 12:26:59,985 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19153.29 MB 2025-02-14 12:26:59,985 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-14 12:26:59,985 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17204.22 MB 2025-02-14 12:26:59,987 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 12:26:59,987 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 12:26:59,987 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.06 seconds 2025-02-14 12:26:59,987 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:26:59,987 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13442.54 MB 2025-02-14 12:26:59,987 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17391.98 MB 2025-02-14 12:26:59,987 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3949.44 MB 2025-02-14 12:26:59,987 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55469.67 MB 2025-02-14 12:26:59,987 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19153.29 MB 2025-02-14 12:26:59,987 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -36316.38 MB 2025-02-14 12:26:59,987 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17391.98 MB 2025-02-14 12:27:00,254 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 12:27:00,254 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 12:27:00,254 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 12:27:00,254 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:27:00,254 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14158.65 MB 2025-02-14 12:27:00,254 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17171.58 MB 2025-02-14 12:27:00,254 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3012.93 MB 2025-02-14 12:27:00,254 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19153.29 MB 2025-02-14 12:27:00,254 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19153.29 MB 2025-02-14 12:27:00,254 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:27:00,254 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17472.84 MB 2025-02-14 12:27:00,272 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8159, cut from 8161 2025-02-14 12:27:00,272 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 12:27:00,278 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 12:27:00,278 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 12:27:00,278 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:27:00,278 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:27:00,278 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17171.58 MB 2025-02-14 12:27:00,278 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25607.17 MB 2025-02-14 12:27:00,278 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8435.59 MB 2025-02-14 12:27:00,278 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19153.29 MB 2025-02-14 12:27:00,278 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29639.05 MB 2025-02-14 12:27:00,278 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10485.76 MB 2025-02-14 12:27:00,278 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25607.17 MB 2025-02-14 12:27:00,429 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7951] 2025-02-14 12:27:00,430 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:27:00,430 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 12:27:00,431 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:27:00,431 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 12:27:00,436 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 12:27:00,437 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:27:00,437 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 12:27:00,437 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 12:28:24,055 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:28:24,055 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 12:28:24,060 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 12:28:24,063 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:28:24,063 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 225, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 12:28:24,064 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:28:24,064 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 225, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 12:28:27,508 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 12:28:27,508 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 12:28:27,508 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.44 seconds 2025-02-14 12:28:27,508 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:28:27,508 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14536.54 MB 2025-02-14 12:28:27,508 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15332.80 MB 2025-02-14 12:28:27,508 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 796.26 MB 2025-02-14 12:28:27,508 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38027.66 MB 2025-02-14 12:28:27,508 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17783.85 MB 2025-02-14 12:28:27,508 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20243.81 MB 2025-02-14 12:28:27,508 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24235.21 MB 2025-02-14 12:28:27,523 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 12:28:27,523 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 12:28:27,523 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:28:27,523 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:28:27,523 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15332.80 MB 2025-02-14 12:28:27,523 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15536.32 MB 2025-02-14 12:28:27,523 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 203.52 MB 2025-02-14 12:28:27,523 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17783.85 MB 2025-02-14 12:28:27,523 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19193.14 MB 2025-02-14 12:28:27,523 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1409.29 MB 2025-02-14 12:28:27,523 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18133.28 MB 2025-02-14 12:28:28,482 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 12:28:28,482 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 12:28:28,482 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.96 seconds 2025-02-14 12:28:28,482 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:28:28,482 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15536.32 MB 2025-02-14 12:28:28,482 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15800.41 MB 2025-02-14 12:28:28,482 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 264.09 MB 2025-02-14 12:28:28,482 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19193.14 MB 2025-02-14 12:28:28,482 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18891.15 MB 2025-02-14 12:28:28,482 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -301.99 MB 2025-02-14 12:28:28,482 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19792.73 MB 2025-02-14 12:28:28,491 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 12:28:28,491 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 12:28:28,491 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:28:28,492 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:28:28,492 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15800.41 MB 2025-02-14 12:28:28,492 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16740.49 MB 2025-02-14 12:28:28,492 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 940.08 MB 2025-02-14 12:28:28,492 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18891.15 MB 2025-02-14 12:28:28,492 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19360.91 MB 2025-02-14 12:28:28,492 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 469.76 MB 2025-02-14 12:28:28,492 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17445.67 MB 2025-02-14 12:28:28,598 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 12:28:28,598 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 12:28:28,598 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 12:28:28,598 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:28:28,598 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16740.49 MB 2025-02-14 12:28:28,598 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17856.11 MB 2025-02-14 12:28:28,598 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1115.62 MB 2025-02-14 12:28:28,598 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19360.91 MB 2025-02-14 12:28:28,598 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22179.48 MB 2025-02-14 12:28:28,598 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2818.57 MB 2025-02-14 12:28:28,598 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20615.41 MB 2025-02-14 12:28:28,599 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 12:28:28,599 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 12:28:28,599 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 12:28:28,599 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:28:28,599 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15800.41 MB 2025-02-14 12:28:28,599 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17856.11 MB 2025-02-14 12:28:28,599 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2055.69 MB 2025-02-14 12:28:28,599 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18891.15 MB 2025-02-14 12:28:28,599 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22179.48 MB 2025-02-14 12:28:28,599 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3288.33 MB 2025-02-14 12:28:28,599 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20615.41 MB 2025-02-14 12:28:28,680 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 12:28:28,680 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 12:28:28,680 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 12:28:28,680 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:28:28,680 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18619.05 MB 2025-02-14 12:28:28,680 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19000.63 MB 2025-02-14 12:28:28,680 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 381.58 MB 2025-02-14 12:28:28,680 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22179.48 MB 2025-02-14 12:28:28,680 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22385.00 MB 2025-02-14 12:28:28,680 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 205.52 MB 2025-02-14 12:28:28,680 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19357.23 MB 2025-02-14 12:28:28,691 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 12:28:28,691 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 12:28:28,691 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:28:28,691 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:28:28,691 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19206.05 MB 2025-02-14 12:28:28,691 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19432.82 MB 2025-02-14 12:28:28,691 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.77 MB 2025-02-14 12:28:28,691 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22385.00 MB 2025-02-14 12:28:28,691 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22385.00 MB 2025-02-14 12:28:28,691 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:28:28,691 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19477.25 MB 2025-02-14 12:28:28,692 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 12:28:28,692 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 12:28:28,692 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.63 seconds 2025-02-14 12:28:28,692 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:28:28,692 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13752.62 MB 2025-02-14 12:28:28,692 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19633.69 MB 2025-02-14 12:28:28,692 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5881.07 MB 2025-02-14 12:28:28,692 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38027.66 MB 2025-02-14 12:28:28,692 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22385.00 MB 2025-02-14 12:28:28,692 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15642.66 MB 2025-02-14 12:28:28,692 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19633.69 MB 2025-02-14 12:28:28,958 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 12:28:28,958 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 12:28:28,958 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 12:28:28,958 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:28:28,958 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14794.30 MB 2025-02-14 12:28:28,958 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17805.78 MB 2025-02-14 12:28:28,958 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3011.48 MB 2025-02-14 12:28:28,958 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22385.00 MB 2025-02-14 12:28:28,958 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22385.00 MB 2025-02-14 12:28:28,958 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:28:28,958 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18106.85 MB 2025-02-14 12:28:28,976 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8154, cut from 8156 2025-02-14 12:28:28,976 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2.'] 2025-02-14 12:28:28,982 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 12:28:28,982 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 12:28:28,982 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:28:28,982 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:28:28,982 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17805.78 MB 2025-02-14 12:28:28,982 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26236.46 MB 2025-02-14 12:28:28,982 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8430.68 MB 2025-02-14 12:28:28,982 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22385.00 MB 2025-02-14 12:28:28,982 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32864.47 MB 2025-02-14 12:28:28,982 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10479.47 MB 2025-02-14 12:28:28,982 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26236.46 MB 2025-02-14 12:28:29,132 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7946] 2025-02-14 12:28:29,133 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:28:29,133 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 12:28:29,134 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:28:29,134 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 12:28:29,139 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 12:28:29,140 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:28:29,140 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 12:28:29,140 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2.'] 2025-02-14 12:30:34,072 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:30:34,073 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 12:30:34,077 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 12:30:34,081 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:30:34,081 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1978, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 12:30:34,082 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:30:34,082 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1978, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 12:31:04,424 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 12:31:04,424 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 12:31:04,424 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.33 seconds 2025-02-14 12:31:04,424 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:31:04,424 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26751.73 MB 2025-02-14 12:31:04,424 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33752.02 MB 2025-02-14 12:31:04,424 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7000.29 MB 2025-02-14 12:31:04,424 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45436.90 MB 2025-02-14 12:31:04,424 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40670.07 MB 2025-02-14 12:31:04,424 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4766.83 MB 2025-02-14 12:31:04,424 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42565.69 MB 2025-02-14 12:31:04,553 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 12:31:04,553 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 12:31:04,553 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 12:31:04,553 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:31:04,553 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33752.02 MB 2025-02-14 12:31:04,553 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26060.86 MB 2025-02-14 12:31:04,553 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7691.16 MB 2025-02-14 12:31:04,553 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40670.07 MB 2025-02-14 12:31:04,553 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 63921.19 MB 2025-02-14 12:31:04,553 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 23251.12 MB 2025-02-14 12:31:04,553 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54080.08 MB 2025-02-14 12:31:06,479 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 12:31:06,479 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 12:31:06,479 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 12:31:06,479 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:31:06,479 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26060.86 MB 2025-02-14 12:31:06,479 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26591.70 MB 2025-02-14 12:31:06,479 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 12:31:06,479 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63921.19 MB 2025-02-14 12:31:06,479 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35085.35 MB 2025-02-14 12:31:06,479 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28835.84 MB 2025-02-14 12:31:06,479 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30571.03 MB 2025-02-14 12:31:06,492 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 12:31:06,492 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 12:31:06,492 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:31:06,492 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:31:06,492 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26591.70 MB 2025-02-14 12:31:06,492 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28481.23 MB 2025-02-14 12:31:06,492 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 12:31:06,492 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35085.35 MB 2025-02-14 12:31:06,492 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35085.35 MB 2025-02-14 12:31:06,492 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:31:06,492 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29898.66 MB 2025-02-14 12:31:06,709 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 12:31:06,710 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 12:31:06,710 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 12:31:06,710 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:31:06,710 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28481.23 MB 2025-02-14 12:31:06,710 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30723.09 MB 2025-02-14 12:31:06,710 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 12:31:06,710 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35085.35 MB 2025-02-14 12:31:06,710 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39803.94 MB 2025-02-14 12:31:06,710 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 12:31:06,710 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36267.37 MB 2025-02-14 12:31:06,710 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 12:31:06,710 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 12:31:06,710 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 12:31:06,710 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:31:06,710 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26591.70 MB 2025-02-14 12:31:06,710 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30723.09 MB 2025-02-14 12:31:06,710 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 12:31:06,710 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35085.35 MB 2025-02-14 12:31:06,710 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39803.94 MB 2025-02-14 12:31:06,710 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 12:31:06,710 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36267.37 MB 2025-02-14 12:31:06,885 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 12:31:06,885 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 12:31:06,885 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 12:31:06,885 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:31:06,885 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32256.63 MB 2025-02-14 12:31:06,885 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33023.63 MB 2025-02-14 12:31:06,885 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 12:31:06,885 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39803.94 MB 2025-02-14 12:31:06,885 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40221.28 MB 2025-02-14 12:31:06,885 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 12:31:06,885 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33731.42 MB 2025-02-14 12:31:06,905 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 12:31:06,905 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 12:31:06,905 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:31:06,905 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:31:06,905 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33436.52 MB 2025-02-14 12:31:06,905 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33664.66 MB 2025-02-14 12:31:06,905 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.14 MB 2025-02-14 12:31:06,905 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40221.28 MB 2025-02-14 12:31:06,905 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40221.28 MB 2025-02-14 12:31:06,905 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:31:06,905 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33885.51 MB 2025-02-14 12:31:06,906 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 12:31:06,906 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 12:31:06,906 - resource_logging.py:150 - __exit__ - DEBUG - Time: 32.82 seconds 2025-02-14 12:31:06,906 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:31:06,906 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19860.22 MB 2025-02-14 12:31:06,906 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33864.52 MB 2025-02-14 12:31:06,906 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14004.31 MB 2025-02-14 12:31:06,906 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45436.90 MB 2025-02-14 12:31:06,906 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40221.28 MB 2025-02-14 12:31:06,906 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5215.62 MB 2025-02-14 12:31:06,906 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33885.51 MB 2025-02-14 12:31:07,174 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 12:31:07,174 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 12:31:07,174 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 12:31:07,174 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:31:07,174 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33864.52 MB 2025-02-14 12:31:07,174 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24845.94 MB 2025-02-14 12:31:07,174 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9018.59 MB 2025-02-14 12:31:07,174 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40221.28 MB 2025-02-14 12:31:07,174 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40221.28 MB 2025-02-14 12:31:07,174 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:31:07,174 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36361.14 MB 2025-02-14 12:31:07,192 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8113, cut from 8115 2025-02-14 12:31:07,192 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 12:31:07,198 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 12:31:07,198 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 12:31:07,198 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:31:07,198 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:31:07,198 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24845.94 MB 2025-02-14 12:31:07,198 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33234.36 MB 2025-02-14 12:31:07,198 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8388.42 MB 2025-02-14 12:31:07,198 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40221.28 MB 2025-02-14 12:31:07,198 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44390.42 MB 2025-02-14 12:31:07,198 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4169.14 MB 2025-02-14 12:31:07,198 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33234.36 MB 2025-02-14 12:31:07,361 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7905] 2025-02-14 12:31:07,362 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:31:07,362 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 12:31:07,363 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:31:07,363 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 12:31:07,368 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 12:31:07,369 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:31:07,369 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 12:31:07,369 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 12:31:15,531 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:31:15,531 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 12:31:15,538 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 12:31:15,543 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:31:15,544 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 3227, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 12:31:15,546 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:31:15,546 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 3227, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 12:32:06,264 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 12:32:06,264 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 12:32:06,264 - resource_logging.py:150 - __exit__ - DEBUG - Time: 50.70 seconds 2025-02-14 12:32:06,264 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:32:06,264 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35455.88 MB 2025-02-14 12:32:06,264 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46876.97 MB 2025-02-14 12:32:06,264 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11421.09 MB 2025-02-14 12:32:06,264 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 79389.79 MB 2025-02-14 12:32:06,264 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50818.19 MB 2025-02-14 12:32:06,264 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28571.60 MB 2025-02-14 12:32:06,264 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 58297.14 MB 2025-02-14 12:32:06,492 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 12:32:06,492 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 12:32:06,492 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 12:32:06,492 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:32:06,492 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46876.97 MB 2025-02-14 12:32:06,492 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32554.48 MB 2025-02-14 12:32:06,492 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -14322.49 MB 2025-02-14 12:32:06,492 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50818.19 MB 2025-02-14 12:32:06,492 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 95206.51 MB 2025-02-14 12:32:06,492 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 44388.32 MB 2025-02-14 12:32:06,492 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 80610.77 MB 2025-02-14 12:32:08,490 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 12:32:08,490 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 12:32:08,490 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.00 seconds 2025-02-14 12:32:08,490 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:32:08,490 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32554.48 MB 2025-02-14 12:32:08,490 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33085.32 MB 2025-02-14 12:32:08,490 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 12:32:08,490 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 95206.51 MB 2025-02-14 12:32:08,490 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35102.13 MB 2025-02-14 12:32:08,490 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -60104.38 MB 2025-02-14 12:32:08,490 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37065.69 MB 2025-02-14 12:32:08,504 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 12:32:08,504 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 12:32:08,504 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:32:08,504 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:32:08,504 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33085.32 MB 2025-02-14 12:32:08,504 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34974.85 MB 2025-02-14 12:32:08,504 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 12:32:08,504 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35102.13 MB 2025-02-14 12:32:08,504 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38405.14 MB 2025-02-14 12:32:08,504 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 12:32:08,504 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36392.28 MB 2025-02-14 12:32:08,714 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 12:32:08,714 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 12:32:08,714 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 12:32:08,714 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:32:08,714 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34974.85 MB 2025-02-14 12:32:08,714 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37216.71 MB 2025-02-14 12:32:08,715 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 12:32:08,715 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38405.14 MB 2025-02-14 12:32:08,715 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45011.17 MB 2025-02-14 12:32:08,715 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 12:32:08,715 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42760.99 MB 2025-02-14 12:32:08,715 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 12:32:08,715 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 12:32:08,715 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 12:32:08,715 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:32:08,715 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33085.32 MB 2025-02-14 12:32:08,715 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37216.71 MB 2025-02-14 12:32:08,715 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 12:32:08,715 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35102.13 MB 2025-02-14 12:32:08,715 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45011.17 MB 2025-02-14 12:32:08,715 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9909.04 MB 2025-02-14 12:32:08,715 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42760.99 MB 2025-02-14 12:32:08,885 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 12:32:08,885 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 12:32:08,885 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 12:32:08,885 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:32:08,885 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38750.25 MB 2025-02-14 12:32:08,885 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39517.25 MB 2025-02-14 12:32:08,885 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 12:32:08,885 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45011.17 MB 2025-02-14 12:32:08,885 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45428.51 MB 2025-02-14 12:32:08,885 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 12:32:08,885 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40225.04 MB 2025-02-14 12:32:08,904 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 12:32:08,905 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 12:32:08,905 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:32:08,905 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:32:08,905 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39930.14 MB 2025-02-14 12:32:08,905 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40158.34 MB 2025-02-14 12:32:08,905 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.20 MB 2025-02-14 12:32:08,905 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45428.51 MB 2025-02-14 12:32:08,905 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45428.51 MB 2025-02-14 12:32:08,905 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:32:08,905 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40396.43 MB 2025-02-14 12:32:08,906 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 12:32:08,906 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 12:32:08,906 - resource_logging.py:150 - __exit__ - DEBUG - Time: 53.36 seconds 2025-02-14 12:32:08,906 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:32:08,906 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24212.29 MB 2025-02-14 12:32:08,906 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40358.45 MB 2025-02-14 12:32:08,906 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 16146.16 MB 2025-02-14 12:32:08,906 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 68144.86 MB 2025-02-14 12:32:08,906 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45428.51 MB 2025-02-14 12:32:08,906 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22716.35 MB 2025-02-14 12:32:08,906 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40396.43 MB 2025-02-14 12:32:09,177 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 12:32:09,177 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 12:32:09,177 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 12:32:09,177 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:32:09,177 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40358.45 MB 2025-02-14 12:32:09,177 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29201.82 MB 2025-02-14 12:32:09,177 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -11156.63 MB 2025-02-14 12:32:09,177 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45428.51 MB 2025-02-14 12:32:09,178 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45428.51 MB 2025-02-14 12:32:09,178 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:32:09,178 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42858.14 MB 2025-02-14 12:32:09,195 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8123, cut from 8125 2025-02-14 12:32:09,196 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 12:32:09,201 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 12:32:09,201 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 12:32:09,201 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:32:09,201 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:32:09,201 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29201.82 MB 2025-02-14 12:32:09,201 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37601.21 MB 2025-02-14 12:32:09,201 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8399.39 MB 2025-02-14 12:32:09,202 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45428.51 MB 2025-02-14 12:32:09,202 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49603.94 MB 2025-02-14 12:32:09,202 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4175.43 MB 2025-02-14 12:32:09,202 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37601.21 MB 2025-02-14 12:32:09,364 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7915] 2025-02-14 12:32:09,366 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:32:09,366 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 12:32:09,367 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:32:09,367 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 12:32:09,371 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 12:32:09,372 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:32:09,373 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 12:32:09,373 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 12:33:05,980 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:33:05,980 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 12:33:05,985 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 12:33:05,989 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:33:05,989 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 130, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 12:33:05,990 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:33:05,990 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 130, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 12:33:08,044 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 12:33:08,044 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 12:33:08,044 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.05 seconds 2025-02-14 12:33:08,045 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:33:08,045 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13874.57 MB 2025-02-14 12:33:08,045 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14334.63 MB 2025-02-14 12:33:08,045 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 460.06 MB 2025-02-14 12:33:08,045 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57954.80 MB 2025-02-14 12:33:08,045 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 16909.34 MB 2025-02-14 12:33:08,045 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -41045.46 MB 2025-02-14 12:33:08,045 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23346.74 MB 2025-02-14 12:33:08,055 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 12:33:08,055 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 12:33:08,055 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:33:08,055 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:33:08,055 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14334.63 MB 2025-02-14 12:33:08,055 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14557.53 MB 2025-02-14 12:33:08,056 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 222.90 MB 2025-02-14 12:33:08,056 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 16909.34 MB 2025-02-14 12:33:08,056 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17370.71 MB 2025-02-14 12:33:08,056 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 461.37 MB 2025-02-14 12:33:08,056 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16167.76 MB 2025-02-14 12:33:08,700 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 12:33:08,700 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 12:33:08,701 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.64 seconds 2025-02-14 12:33:08,701 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:33:08,701 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14557.53 MB 2025-02-14 12:33:08,701 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14730.05 MB 2025-02-14 12:33:08,701 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 172.52 MB 2025-02-14 12:33:08,701 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17370.71 MB 2025-02-14 12:33:08,701 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17370.71 MB 2025-02-14 12:33:08,701 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:33:08,701 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18729.00 MB 2025-02-14 12:33:08,707 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 12:33:08,707 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 12:33:08,707 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 12:33:08,707 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:33:08,707 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14729.99 MB 2025-02-14 12:33:08,707 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15343.94 MB 2025-02-14 12:33:08,707 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 613.95 MB 2025-02-14 12:33:08,707 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17370.71 MB 2025-02-14 12:33:08,707 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17370.71 MB 2025-02-14 12:33:08,707 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:33:08,707 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15804.61 MB 2025-02-14 12:33:08,777 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 12:33:08,777 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 12:33:08,777 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 12:33:08,777 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:33:08,777 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15343.94 MB 2025-02-14 12:33:08,777 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16072.58 MB 2025-02-14 12:33:08,777 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 728.65 MB 2025-02-14 12:33:08,777 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17370.71 MB 2025-02-14 12:33:08,777 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18603.84 MB 2025-02-14 12:33:08,777 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1233.13 MB 2025-02-14 12:33:08,777 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17874.43 MB 2025-02-14 12:33:08,778 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 12:33:08,778 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 12:33:08,778 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 12:33:08,778 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:33:08,778 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14729.99 MB 2025-02-14 12:33:08,778 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16072.58 MB 2025-02-14 12:33:08,778 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1342.60 MB 2025-02-14 12:33:08,778 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17370.71 MB 2025-02-14 12:33:08,778 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18603.84 MB 2025-02-14 12:33:08,778 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1233.13 MB 2025-02-14 12:33:08,778 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17874.43 MB 2025-02-14 12:33:08,832 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 12:33:08,832 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 12:33:08,832 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-14 12:33:08,832 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:33:08,832 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16570.98 MB 2025-02-14 12:33:08,832 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16820.26 MB 2025-02-14 12:33:08,832 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 249.28 MB 2025-02-14 12:33:08,832 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18603.84 MB 2025-02-14 12:33:08,832 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18735.96 MB 2025-02-14 12:33:08,832 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 132.12 MB 2025-02-14 12:33:08,832 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17062.79 MB 2025-02-14 12:33:08,840 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 12:33:08,840 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 12:33:08,840 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:33:08,840 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:33:08,840 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16954.46 MB 2025-02-14 12:33:08,840 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17151.03 MB 2025-02-14 12:33:08,840 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 196.57 MB 2025-02-14 12:33:08,840 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18735.96 MB 2025-02-14 12:33:08,840 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18740.15 MB 2025-02-14 12:33:08,840 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-14 12:33:08,840 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17151.03 MB 2025-02-14 12:33:08,841 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 12:33:08,841 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 12:33:08,841 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.85 seconds 2025-02-14 12:33:08,841 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:33:08,841 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13421.64 MB 2025-02-14 12:33:08,841 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17343.27 MB 2025-02-14 12:33:08,841 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3921.64 MB 2025-02-14 12:33:08,841 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57954.80 MB 2025-02-14 12:33:08,841 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18740.15 MB 2025-02-14 12:33:08,841 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -39214.65 MB 2025-02-14 12:33:08,841 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17343.27 MB 2025-02-14 12:33:09,101 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 12:33:09,101 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 12:33:09,101 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 12:33:09,101 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:33:09,101 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17343.27 MB 2025-02-14 12:33:09,101 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17015.06 MB 2025-02-14 12:33:09,101 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -328.21 MB 2025-02-14 12:33:09,101 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18740.15 MB 2025-02-14 12:33:09,101 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19130.22 MB 2025-02-14 12:33:09,101 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 390.07 MB 2025-02-14 12:33:09,101 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18688.05 MB 2025-02-14 12:33:09,118 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 7803, cut from 7805 2025-02-14 12:33:09,119 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 12:33:09,124 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 12:33:09,125 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 12:33:09,125 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:33:09,125 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:33:09,125 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17015.06 MB 2025-02-14 12:33:09,125 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25083.19 MB 2025-02-14 12:33:09,125 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8068.13 MB 2025-02-14 12:33:09,125 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19130.22 MB 2025-02-14 12:33:09,125 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29160.90 MB 2025-02-14 12:33:09,125 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10030.68 MB 2025-02-14 12:33:09,125 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25083.19 MB 2025-02-14 12:33:09,269 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7595] 2025-02-14 12:33:09,270 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:33:09,270 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 12:33:09,271 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:33:09,271 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 12:33:09,276 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 12:33:09,277 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:33:09,277 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 12:33:09,277 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 12:33:32,803 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:33:32,803 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 12:33:32,808 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 12:33:32,811 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:33:32,811 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1217, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 12:33:32,812 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:33:32,812 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1217, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 12:33:51,592 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 12:33:51,592 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 12:33:51,592 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.77 seconds 2025-02-14 12:33:51,592 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:33:51,592 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21448.96 MB 2025-02-14 12:33:51,592 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25756.51 MB 2025-02-14 12:33:51,592 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4307.55 MB 2025-02-14 12:33:51,592 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37184.60 MB 2025-02-14 12:33:51,592 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37264.29 MB 2025-02-14 12:33:51,592 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 79.69 MB 2025-02-14 12:33:51,592 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34770.70 MB 2025-02-14 12:33:51,660 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 12:33:51,660 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 12:33:51,660 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 12:33:51,660 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:33:51,660 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25756.51 MB 2025-02-14 12:33:51,660 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22104.65 MB 2025-02-14 12:33:51,660 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3651.85 MB 2025-02-14 12:33:51,660 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37264.29 MB 2025-02-14 12:33:51,660 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45539.66 MB 2025-02-14 12:33:51,660 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8275.36 MB 2025-02-14 12:33:51,660 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38258.69 MB 2025-02-14 12:33:53,576 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 12:33:53,576 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 12:33:53,576 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 12:33:53,576 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:33:53,576 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22104.65 MB 2025-02-14 12:33:53,576 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22635.50 MB 2025-02-14 12:33:53,576 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 12:33:53,576 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45539.66 MB 2025-02-14 12:33:53,576 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28944.89 MB 2025-02-14 12:33:53,576 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16594.76 MB 2025-02-14 12:33:53,576 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26614.83 MB 2025-02-14 12:33:53,589 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 12:33:53,589 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 12:33:53,589 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:33:53,589 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:33:53,589 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22635.50 MB 2025-02-14 12:33:53,589 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24525.03 MB 2025-02-14 12:33:53,589 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 12:33:53,589 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28944.89 MB 2025-02-14 12:33:53,589 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28944.89 MB 2025-02-14 12:33:53,589 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:33:53,589 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25942.46 MB 2025-02-14 12:33:53,795 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 12:33:53,795 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 12:33:53,795 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 12:33:53,795 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:33:53,795 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24525.03 MB 2025-02-14 12:33:53,795 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26766.89 MB 2025-02-14 12:33:53,795 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 12:33:53,795 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28944.89 MB 2025-02-14 12:33:53,795 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34607.20 MB 2025-02-14 12:33:53,795 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 12:33:53,795 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32311.17 MB 2025-02-14 12:33:53,796 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 12:33:53,796 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 12:33:53,796 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 12:33:53,796 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:33:53,796 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22635.50 MB 2025-02-14 12:33:53,796 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26766.89 MB 2025-02-14 12:33:53,796 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 12:33:53,796 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28944.89 MB 2025-02-14 12:33:53,796 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34607.20 MB 2025-02-14 12:33:53,796 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 12:33:53,796 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32311.17 MB 2025-02-14 12:33:53,958 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 12:33:53,958 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 12:33:53,958 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 12:33:53,958 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:33:53,958 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28300.43 MB 2025-02-14 12:33:53,958 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29067.43 MB 2025-02-14 12:33:53,958 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 12:33:53,958 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34607.20 MB 2025-02-14 12:33:53,958 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35022.44 MB 2025-02-14 12:33:53,958 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 12:33:53,958 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29775.22 MB 2025-02-14 12:33:53,977 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 12:33:53,977 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 12:33:53,977 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:33:53,977 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:33:53,977 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29480.32 MB 2025-02-14 12:33:53,977 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29709.44 MB 2025-02-14 12:33:53,977 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.12 MB 2025-02-14 12:33:53,977 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35022.44 MB 2025-02-14 12:33:53,977 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35022.44 MB 2025-02-14 12:33:53,977 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:33:53,977 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29920.71 MB 2025-02-14 12:33:53,978 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 12:33:53,978 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 12:33:53,978 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.16 seconds 2025-02-14 12:33:53,978 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:33:53,978 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17208.83 MB 2025-02-14 12:33:53,978 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29910.36 MB 2025-02-14 12:33:53,978 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12701.53 MB 2025-02-14 12:33:53,978 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37184.60 MB 2025-02-14 12:33:53,978 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35022.44 MB 2025-02-14 12:33:53,978 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2162.16 MB 2025-02-14 12:33:53,978 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29920.71 MB 2025-02-14 12:33:54,245 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 12:33:54,245 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 12:33:54,245 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 12:33:54,246 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:33:54,246 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29910.36 MB 2025-02-14 12:33:54,246 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22210.94 MB 2025-02-14 12:33:54,246 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7699.43 MB 2025-02-14 12:33:54,246 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35022.44 MB 2025-02-14 12:33:54,246 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35022.44 MB 2025-02-14 12:33:54,246 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:33:54,246 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32420.19 MB 2025-02-14 12:33:54,264 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8156, cut from 8158 2025-02-14 12:33:54,264 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 12:33:54,270 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 12:33:54,270 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 12:33:54,270 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:33:54,270 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:33:54,270 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22210.94 MB 2025-02-14 12:33:54,270 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30644.23 MB 2025-02-14 12:33:54,270 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8433.30 MB 2025-02-14 12:33:54,270 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35022.44 MB 2025-02-14 12:33:54,270 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43406.85 MB 2025-02-14 12:33:54,270 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8384.41 MB 2025-02-14 12:33:54,270 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30644.23 MB 2025-02-14 12:33:54,427 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7948] 2025-02-14 12:33:54,428 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:33:54,428 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 12:33:54,429 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:33:54,429 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 12:33:54,434 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 12:33:54,435 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:33:54,435 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 12:33:54,435 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 12:35:02,229 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:35:02,229 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 12:35:02,234 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 12:35:02,238 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:35:02,238 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 474, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 12:35:02,239 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:35:02,239 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 474, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 12:35:09,547 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 12:35:09,547 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 12:35:09,547 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.30 seconds 2025-02-14 12:35:09,547 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:35:09,547 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16271.61 MB 2025-02-14 12:35:09,547 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17949.34 MB 2025-02-14 12:35:09,547 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1677.72 MB 2025-02-14 12:35:09,547 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51791.27 MB 2025-02-14 12:35:09,547 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22361.93 MB 2025-02-14 12:35:09,547 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29429.33 MB 2025-02-14 12:35:09,547 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26876.25 MB 2025-02-14 12:35:09,581 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 12:35:09,581 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 12:35:09,581 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 12:35:09,581 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:35:09,581 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17949.34 MB 2025-02-14 12:35:09,581 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18243.08 MB 2025-02-14 12:35:09,581 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 293.74 MB 2025-02-14 12:35:09,581 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22361.93 MB 2025-02-14 12:35:09,581 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28357.69 MB 2025-02-14 12:35:09,581 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5995.76 MB 2025-02-14 12:35:09,581 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25490.70 MB 2025-02-14 12:35:11,489 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 12:35:11,489 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 12:35:11,489 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 12:35:11,489 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:35:11,489 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18243.08 MB 2025-02-14 12:35:11,489 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18773.92 MB 2025-02-14 12:35:11,489 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 12:35:11,489 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28357.69 MB 2025-02-14 12:35:11,489 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21864.91 MB 2025-02-14 12:35:11,489 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6492.78 MB 2025-02-14 12:35:11,489 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22754.29 MB 2025-02-14 12:35:11,503 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 12:35:11,503 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 12:35:11,503 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:35:11,503 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:35:11,503 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18773.92 MB 2025-02-14 12:35:11,503 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20663.45 MB 2025-02-14 12:35:11,503 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 12:35:11,503 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21864.91 MB 2025-02-14 12:35:11,503 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24696.06 MB 2025-02-14 12:35:11,503 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-14 12:35:11,503 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22080.88 MB 2025-02-14 12:35:11,709 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 12:35:11,709 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 12:35:11,709 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 12:35:11,709 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:35:11,709 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20663.45 MB 2025-02-14 12:35:11,709 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22905.31 MB 2025-02-14 12:35:11,709 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 12:35:11,709 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24696.06 MB 2025-02-14 12:35:11,709 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30830.23 MB 2025-02-14 12:35:11,709 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 12:35:11,709 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28449.59 MB 2025-02-14 12:35:11,710 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 12:35:11,710 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 12:35:11,710 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 12:35:11,710 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:35:11,710 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18773.92 MB 2025-02-14 12:35:11,710 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22905.31 MB 2025-02-14 12:35:11,710 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 12:35:11,710 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21864.91 MB 2025-02-14 12:35:11,710 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30830.23 MB 2025-02-14 12:35:11,710 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8965.32 MB 2025-02-14 12:35:11,710 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28449.59 MB 2025-02-14 12:35:11,872 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 12:35:11,872 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 12:35:11,872 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 12:35:11,872 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:35:11,872 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24438.85 MB 2025-02-14 12:35:11,872 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25205.85 MB 2025-02-14 12:35:11,872 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 12:35:11,872 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30830.23 MB 2025-02-14 12:35:11,872 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31245.47 MB 2025-02-14 12:35:11,872 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 12:35:11,872 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25913.64 MB 2025-02-14 12:35:11,890 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 12:35:11,890 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 12:35:11,890 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:35:11,891 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:35:11,891 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25618.74 MB 2025-02-14 12:35:11,891 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25848.51 MB 2025-02-14 12:35:11,891 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.77 MB 2025-02-14 12:35:11,891 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31245.47 MB 2025-02-14 12:35:11,891 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31245.47 MB 2025-02-14 12:35:11,891 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:35:11,891 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26070.63 MB 2025-02-14 12:35:11,892 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 12:35:11,892 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 12:35:11,892 - resource_logging.py:150 - __exit__ - DEBUG - Time: 9.65 seconds 2025-02-14 12:35:11,892 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:35:11,892 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14620.16 MB 2025-02-14 12:35:11,892 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26049.59 MB 2025-02-14 12:35:11,892 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11429.43 MB 2025-02-14 12:35:11,892 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51791.27 MB 2025-02-14 12:35:11,892 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31245.47 MB 2025-02-14 12:35:11,892 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20545.80 MB 2025-02-14 12:35:11,892 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26070.63 MB 2025-02-14 12:35:12,160 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 12:35:12,160 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 12:35:12,160 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 12:35:12,160 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:35:12,160 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26049.59 MB 2025-02-14 12:35:12,160 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19624.55 MB 2025-02-14 12:35:12,160 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6425.04 MB 2025-02-14 12:35:12,160 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31245.47 MB 2025-02-14 12:35:12,160 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31245.47 MB 2025-02-14 12:35:12,160 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:35:12,160 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28561.25 MB 2025-02-14 12:35:12,178 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 12:35:12,179 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 12:35:12,185 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 12:35:12,185 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 12:35:12,185 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:35:12,185 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:35:12,185 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19624.55 MB 2025-02-14 12:35:12,185 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28063.57 MB 2025-02-14 12:35:12,185 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 12:35:12,185 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31245.47 MB 2025-02-14 12:35:12,185 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41735.42 MB 2025-02-14 12:35:12,185 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 12:35:12,185 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28063.57 MB 2025-02-14 12:35:12,338 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 12:35:12,339 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:35:12,339 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 12:35:12,340 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:35:12,340 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 12:35:12,345 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 12:35:12,346 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:35:12,346 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 12:35:12,346 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 12:36:29,384 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:36:29,384 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 12:36:29,388 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 12:36:29,392 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:36:29,392 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1351, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 12:36:29,393 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:36:29,393 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1351, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 12:36:50,093 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 12:36:50,093 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 12:36:50,093 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.69 seconds 2025-02-14 12:36:50,093 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:36:50,093 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22382.69 MB 2025-02-14 12:36:50,093 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27164.20 MB 2025-02-14 12:36:50,093 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4781.51 MB 2025-02-14 12:36:50,093 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54320.43 MB 2025-02-14 12:36:50,093 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38468.06 MB 2025-02-14 12:36:50,093 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15852.37 MB 2025-02-14 12:36:50,093 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36157.42 MB 2025-02-14 12:36:50,167 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 12:36:50,167 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 12:36:50,167 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 12:36:50,167 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:36:50,167 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27164.20 MB 2025-02-14 12:36:50,167 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22801.28 MB 2025-02-14 12:36:50,167 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4362.92 MB 2025-02-14 12:36:50,167 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38468.06 MB 2025-02-14 12:36:50,167 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47630.52 MB 2025-02-14 12:36:50,167 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9162.46 MB 2025-02-14 12:36:50,167 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40962.87 MB 2025-02-14 12:36:52,071 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 12:36:52,071 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 12:36:52,071 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.90 seconds 2025-02-14 12:36:52,071 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:36:52,071 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22801.28 MB 2025-02-14 12:36:52,071 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23332.12 MB 2025-02-14 12:36:52,071 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 12:36:52,071 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47630.52 MB 2025-02-14 12:36:52,071 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33686.55 MB 2025-02-14 12:36:52,071 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13943.96 MB 2025-02-14 12:36:52,071 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27311.45 MB 2025-02-14 12:36:52,085 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 12:36:52,085 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 12:36:52,085 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:36:52,085 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:36:52,085 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23332.12 MB 2025-02-14 12:36:52,085 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25221.65 MB 2025-02-14 12:36:52,085 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 12:36:52,085 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33686.55 MB 2025-02-14 12:36:52,085 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33686.55 MB 2025-02-14 12:36:52,085 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:36:52,085 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26639.08 MB 2025-02-14 12:36:52,294 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 12:36:52,294 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 12:36:52,294 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 12:36:52,294 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:36:52,294 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25221.65 MB 2025-02-14 12:36:52,294 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27463.51 MB 2025-02-14 12:36:52,294 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 12:36:52,294 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33686.55 MB 2025-02-14 12:36:52,294 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37461.43 MB 2025-02-14 12:36:52,294 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 12:36:52,294 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33007.79 MB 2025-02-14 12:36:52,295 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 12:36:52,295 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 12:36:52,295 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 12:36:52,295 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:36:52,295 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23332.12 MB 2025-02-14 12:36:52,295 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27463.51 MB 2025-02-14 12:36:52,295 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 12:36:52,295 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33686.55 MB 2025-02-14 12:36:52,295 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37461.43 MB 2025-02-14 12:36:52,295 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 12:36:52,295 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33007.79 MB 2025-02-14 12:36:52,456 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 12:36:52,456 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 12:36:52,456 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 12:36:52,456 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:36:52,456 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28997.05 MB 2025-02-14 12:36:52,456 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29764.05 MB 2025-02-14 12:36:52,456 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 12:36:52,456 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37461.43 MB 2025-02-14 12:36:52,456 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37878.76 MB 2025-02-14 12:36:52,456 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 12:36:52,456 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30471.84 MB 2025-02-14 12:36:52,474 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 12:36:52,474 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 12:36:52,474 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:36:52,474 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:36:52,474 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30176.94 MB 2025-02-14 12:36:52,474 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30405.37 MB 2025-02-14 12:36:52,475 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.43 MB 2025-02-14 12:36:52,475 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37878.76 MB 2025-02-14 12:36:52,475 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37878.76 MB 2025-02-14 12:36:52,475 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:36:52,475 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30628.22 MB 2025-02-14 12:36:52,476 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 12:36:52,476 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 12:36:52,476 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.08 seconds 2025-02-14 12:36:52,476 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:36:52,476 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17675.70 MB 2025-02-14 12:36:52,476 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30605.46 MB 2025-02-14 12:36:52,476 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12929.77 MB 2025-02-14 12:36:52,476 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54320.43 MB 2025-02-14 12:36:52,476 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37878.76 MB 2025-02-14 12:36:52,476 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16441.67 MB 2025-02-14 12:36:52,476 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30628.22 MB 2025-02-14 12:36:52,743 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 12:36:52,743 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 12:36:52,743 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 12:36:52,743 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:36:52,743 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30605.46 MB 2025-02-14 12:36:52,743 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22664.85 MB 2025-02-14 12:36:52,743 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7940.61 MB 2025-02-14 12:36:52,743 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37878.76 MB 2025-02-14 12:36:52,743 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37878.76 MB 2025-02-14 12:36:52,743 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:36:52,743 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33104.84 MB 2025-02-14 12:36:52,761 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8122, cut from 8124 2025-02-14 12:36:52,761 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 12:36:52,767 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 12:36:52,767 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 12:36:52,767 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:36:52,767 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:36:52,767 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22664.85 MB 2025-02-14 12:36:52,767 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31062.25 MB 2025-02-14 12:36:52,767 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8397.40 MB 2025-02-14 12:36:52,767 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37878.76 MB 2025-02-14 12:36:52,767 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46229.62 MB 2025-02-14 12:36:52,767 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8350.86 MB 2025-02-14 12:36:52,767 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31062.25 MB 2025-02-14 12:36:52,921 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7914] 2025-02-14 12:36:52,922 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:36:52,922 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 12:36:52,923 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:36:52,923 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 12:36:52,928 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 12:36:52,929 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:36:52,929 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 12:36:52,929 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 12:36:59,967 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:36:59,967 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 12:36:59,975 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 12:36:59,981 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:36:59,981 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2218, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 12:36:59,983 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:36:59,983 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2218, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 12:37:34,698 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 12:37:34,699 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 12:37:34,699 - resource_logging.py:150 - __exit__ - DEBUG - Time: 34.70 seconds 2025-02-14 12:37:34,699 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:37:34,699 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28424.09 MB 2025-02-14 12:37:34,699 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36273.73 MB 2025-02-14 12:37:34,699 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7849.64 MB 2025-02-14 12:37:34,699 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54580.48 MB 2025-02-14 12:37:34,699 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41458.60 MB 2025-02-14 12:37:34,699 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13121.88 MB 2025-02-14 12:37:34,699 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45144.02 MB 2025-02-14 12:37:34,808 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 12:37:34,808 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 12:37:34,808 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 12:37:34,808 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:37:34,808 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36273.73 MB 2025-02-14 12:37:34,808 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27308.54 MB 2025-02-14 12:37:34,808 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8965.18 MB 2025-02-14 12:37:34,808 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41458.60 MB 2025-02-14 12:37:34,808 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55253.66 MB 2025-02-14 12:37:34,808 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 13795.07 MB 2025-02-14 12:37:34,808 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48505.83 MB 2025-02-14 12:37:36,740 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 12:37:36,740 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 12:37:36,740 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 12:37:36,740 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:37:36,740 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27308.54 MB 2025-02-14 12:37:36,740 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27839.38 MB 2025-02-14 12:37:36,740 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 12:37:36,740 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55253.66 MB 2025-02-14 12:37:36,740 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30849.11 MB 2025-02-14 12:37:36,740 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24404.56 MB 2025-02-14 12:37:36,740 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31818.72 MB 2025-02-14 12:37:36,754 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 12:37:36,754 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 12:37:36,754 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:37:36,754 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:37:36,754 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27839.38 MB 2025-02-14 12:37:36,754 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29728.92 MB 2025-02-14 12:37:36,754 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 12:37:36,755 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30849.11 MB 2025-02-14 12:37:36,755 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34152.12 MB 2025-02-14 12:37:36,755 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 12:37:36,755 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31146.35 MB 2025-02-14 12:37:36,970 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 12:37:36,970 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 12:37:36,970 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 12:37:36,970 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:37:36,970 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29728.92 MB 2025-02-14 12:37:36,970 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31970.77 MB 2025-02-14 12:37:36,970 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 12:37:36,970 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34152.12 MB 2025-02-14 12:37:36,970 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40286.29 MB 2025-02-14 12:37:36,970 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 12:37:36,971 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37515.05 MB 2025-02-14 12:37:36,971 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 12:37:36,971 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 12:37:36,971 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 12:37:36,971 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:37:36,971 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27839.38 MB 2025-02-14 12:37:36,971 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31970.77 MB 2025-02-14 12:37:36,971 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 12:37:36,971 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30849.11 MB 2025-02-14 12:37:36,971 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40286.29 MB 2025-02-14 12:37:36,971 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9437.18 MB 2025-02-14 12:37:36,971 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37515.05 MB 2025-02-14 12:37:37,143 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 12:37:37,143 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 12:37:37,143 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 12:37:37,143 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:37:37,143 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33504.31 MB 2025-02-14 12:37:37,143 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34271.32 MB 2025-02-14 12:37:37,143 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 12:37:37,143 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40286.29 MB 2025-02-14 12:37:37,143 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40703.62 MB 2025-02-14 12:37:37,143 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 12:37:37,143 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34979.11 MB 2025-02-14 12:37:37,163 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 12:37:37,163 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 12:37:37,163 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:37:37,163 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:37:37,163 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34684.21 MB 2025-02-14 12:37:37,163 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34910.57 MB 2025-02-14 12:37:37,163 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.36 MB 2025-02-14 12:37:37,163 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40703.62 MB 2025-02-14 12:37:37,163 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40703.62 MB 2025-02-14 12:37:37,163 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:37:37,163 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35132.95 MB 2025-02-14 12:37:37,164 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 12:37:37,164 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 12:37:37,164 - resource_logging.py:150 - __exit__ - DEBUG - Time: 37.18 seconds 2025-02-14 12:37:37,164 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:37:37,164 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20696.40 MB 2025-02-14 12:37:37,164 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35111.64 MB 2025-02-14 12:37:37,164 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14415.25 MB 2025-02-14 12:37:37,164 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54580.48 MB 2025-02-14 12:37:37,164 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40703.62 MB 2025-02-14 12:37:37,164 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13876.85 MB 2025-02-14 12:37:37,164 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35132.95 MB 2025-02-14 12:37:37,435 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 12:37:37,435 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 12:37:37,435 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 12:37:37,435 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:37:37,435 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35111.64 MB 2025-02-14 12:37:37,435 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25700.79 MB 2025-02-14 12:37:37,435 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9410.86 MB 2025-02-14 12:37:37,435 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40703.62 MB 2025-02-14 12:37:37,435 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40703.62 MB 2025-02-14 12:37:37,435 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:37:37,435 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37623.31 MB 2025-02-14 12:37:37,453 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 12:37:37,453 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 12:37:37,459 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 12:37:37,459 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 12:37:37,459 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:37:37,460 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:37:37,460 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25700.79 MB 2025-02-14 12:37:37,460 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34139.81 MB 2025-02-14 12:37:37,460 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 12:37:37,460 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40703.62 MB 2025-02-14 12:37:37,460 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49094.33 MB 2025-02-14 12:37:37,460 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 12:37:37,460 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34139.81 MB 2025-02-14 12:37:37,623 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 12:37:37,624 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:37:37,624 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 12:37:37,625 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:37:37,625 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 12:37:37,630 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 12:37:37,631 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:37:37,631 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 12:37:37,631 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 12:38:55,612 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:38:55,613 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 12:38:55,617 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 12:38:55,621 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:38:55,621 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 93, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 12:38:55,622 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:38:55,622 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 93, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 12:38:57,074 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 12:38:57,074 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 12:38:57,074 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.45 seconds 2025-02-14 12:38:57,074 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:38:57,074 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13616.75 MB 2025-02-14 12:38:57,074 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 13945.87 MB 2025-02-14 12:38:57,074 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 329.12 MB 2025-02-14 12:38:57,074 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61679.34 MB 2025-02-14 12:38:57,074 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25732.06 MB 2025-02-14 12:38:57,074 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35947.28 MB 2025-02-14 12:38:57,074 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22861.62 MB 2025-02-14 12:38:57,077 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 12:38:57,077 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 12:38:57,077 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 12:38:57,077 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:38:57,077 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13945.87 MB 2025-02-14 12:38:57,077 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14105.33 MB 2025-02-14 12:38:57,077 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 159.46 MB 2025-02-14 12:38:57,077 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25732.06 MB 2025-02-14 12:38:57,077 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25732.06 MB 2025-02-14 12:38:57,077 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:38:57,077 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14599.07 MB 2025-02-14 12:38:57,527 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 12:38:57,527 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 12:38:57,527 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.45 seconds 2025-02-14 12:38:57,527 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:38:57,527 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14105.33 MB 2025-02-14 12:38:57,527 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14228.75 MB 2025-02-14 12:38:57,527 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 123.42 MB 2025-02-14 12:38:57,527 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25732.06 MB 2025-02-14 12:38:57,527 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25260.20 MB 2025-02-14 12:38:57,527 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -471.86 MB 2025-02-14 12:38:57,527 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18190.83 MB 2025-02-14 12:38:57,533 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 12:38:57,533 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 12:38:57,533 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 12:38:57,533 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:38:57,533 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14228.68 MB 2025-02-14 12:38:57,533 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14667.89 MB 2025-02-14 12:38:57,533 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 439.21 MB 2025-02-14 12:38:57,533 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25260.20 MB 2025-02-14 12:38:57,533 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25260.20 MB 2025-02-14 12:38:57,533 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:38:57,533 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14997.45 MB 2025-02-14 12:38:57,625 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 12:38:57,625 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 12:38:57,625 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 12:38:57,625 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:38:57,625 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14667.89 MB 2025-02-14 12:38:57,625 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15201.36 MB 2025-02-14 12:38:57,625 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 533.47 MB 2025-02-14 12:38:57,625 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25260.20 MB 2025-02-14 12:38:57,625 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25260.20 MB 2025-02-14 12:38:57,625 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:38:57,625 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16478.17 MB 2025-02-14 12:38:57,626 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 12:38:57,626 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 12:38:57,626 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 12:38:57,626 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:38:57,626 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14228.68 MB 2025-02-14 12:38:57,626 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15201.36 MB 2025-02-14 12:38:57,626 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 972.68 MB 2025-02-14 12:38:57,626 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25260.20 MB 2025-02-14 12:38:57,626 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25260.20 MB 2025-02-14 12:38:57,626 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:38:57,626 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16478.17 MB 2025-02-14 12:38:57,672 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 12:38:57,672 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 12:38:57,672 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-14 12:38:57,673 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:38:57,673 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15716.38 MB 2025-02-14 12:38:57,673 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15940.42 MB 2025-02-14 12:38:57,673 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 224.04 MB 2025-02-14 12:38:57,673 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25260.20 MB 2025-02-14 12:38:57,673 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25400.71 MB 2025-02-14 12:38:57,673 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 140.51 MB 2025-02-14 12:38:57,673 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16104.98 MB 2025-02-14 12:38:57,679 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 12:38:57,679 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 12:38:57,679 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 12:38:57,680 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:38:57,680 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16082.13 MB 2025-02-14 12:38:57,680 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16303.66 MB 2025-02-14 12:38:57,680 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 221.52 MB 2025-02-14 12:38:57,680 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25400.71 MB 2025-02-14 12:38:57,680 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25400.71 MB 2025-02-14 12:38:57,680 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:38:57,680 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16303.66 MB 2025-02-14 12:38:57,681 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 12:38:57,681 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 12:38:57,681 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.06 seconds 2025-02-14 12:38:57,681 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:38:57,681 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13292.73 MB 2025-02-14 12:38:57,681 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16501.61 MB 2025-02-14 12:38:57,681 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3208.88 MB 2025-02-14 12:38:57,681 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61679.34 MB 2025-02-14 12:38:57,681 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25400.71 MB 2025-02-14 12:38:57,681 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -36278.63 MB 2025-02-14 12:38:57,681 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16501.61 MB 2025-02-14 12:38:57,946 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 12:38:57,946 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 12:38:57,946 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 12:38:57,946 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:38:57,946 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13832.17 MB 2025-02-14 12:38:57,946 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16799.39 MB 2025-02-14 12:38:57,946 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2967.22 MB 2025-02-14 12:38:57,946 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25400.71 MB 2025-02-14 12:38:57,946 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25400.71 MB 2025-02-14 12:38:57,946 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:38:57,946 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17096.07 MB 2025-02-14 12:38:57,963 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8035, cut from 8037 2025-02-14 12:38:57,964 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 12:38:57,970 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 12:38:57,970 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 12:38:57,970 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:38:57,970 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:38:57,970 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16799.39 MB 2025-02-14 12:38:57,970 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25107.46 MB 2025-02-14 12:38:57,970 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8308.08 MB 2025-02-14 12:38:57,970 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25400.71 MB 2025-02-14 12:38:57,970 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33661.39 MB 2025-02-14 12:38:57,970 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8260.68 MB 2025-02-14 12:38:57,970 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25107.46 MB 2025-02-14 12:38:58,120 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7827] 2025-02-14 12:38:58,122 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:38:58,122 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 12:38:58,123 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:38:58,123 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 12:38:58,127 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 12:38:58,128 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:38:58,128 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 12:38:58,128 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 12:39:06,199 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:39:06,199 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 12:39:06,206 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 12:39:06,212 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:39:06,213 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1960, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 12:39:06,214 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:39:06,214 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1960, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 12:39:36,593 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 12:39:36,593 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 12:39:36,593 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.37 seconds 2025-02-14 12:39:36,593 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:39:36,593 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26626.30 MB 2025-02-14 12:39:36,593 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33563.68 MB 2025-02-14 12:39:36,593 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6937.38 MB 2025-02-14 12:39:36,593 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46051.36 MB 2025-02-14 12:39:36,593 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40454.06 MB 2025-02-14 12:39:36,593 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5597.30 MB 2025-02-14 12:39:36,593 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42440.27 MB 2025-02-14 12:39:36,711 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 12:39:36,711 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 12:39:36,711 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 12:39:36,711 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:39:36,711 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33563.68 MB 2025-02-14 12:39:36,711 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25967.28 MB 2025-02-14 12:39:36,711 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7596.40 MB 2025-02-14 12:39:36,711 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40454.06 MB 2025-02-14 12:39:36,711 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 63350.77 MB 2025-02-14 12:39:36,711 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 22896.71 MB 2025-02-14 12:39:36,711 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 53620.56 MB 2025-02-14 12:39:38,656 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 12:39:38,656 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 12:39:38,656 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-14 12:39:38,656 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:39:38,657 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25967.28 MB 2025-02-14 12:39:38,657 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26498.12 MB 2025-02-14 12:39:38,657 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 12:39:38,657 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63350.77 MB 2025-02-14 12:39:38,657 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30802.97 MB 2025-02-14 12:39:38,657 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32547.80 MB 2025-02-14 12:39:38,657 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30478.49 MB 2025-02-14 12:39:38,670 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 12:39:38,670 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 12:39:38,670 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:39:38,670 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:39:38,670 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26498.12 MB 2025-02-14 12:39:38,670 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28387.65 MB 2025-02-14 12:39:38,670 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 12:39:38,670 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30802.97 MB 2025-02-14 12:39:38,670 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32690.41 MB 2025-02-14 12:39:38,670 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 12:39:38,670 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29805.08 MB 2025-02-14 12:39:38,874 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 12:39:38,874 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 12:39:38,874 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 12:39:38,874 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:39:38,874 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28387.65 MB 2025-02-14 12:39:38,874 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30629.51 MB 2025-02-14 12:39:38,874 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 12:39:38,874 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32690.41 MB 2025-02-14 12:39:38,874 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38824.57 MB 2025-02-14 12:39:38,874 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 12:39:38,874 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36173.79 MB 2025-02-14 12:39:38,875 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 12:39:38,875 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 12:39:38,875 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 12:39:38,875 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:39:38,875 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26498.12 MB 2025-02-14 12:39:38,875 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30629.51 MB 2025-02-14 12:39:38,875 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 12:39:38,875 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30802.97 MB 2025-02-14 12:39:38,875 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38824.57 MB 2025-02-14 12:39:38,875 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-14 12:39:38,875 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36173.79 MB 2025-02-14 12:39:39,039 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 12:39:39,039 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 12:39:39,039 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 12:39:39,039 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:39:39,039 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32163.05 MB 2025-02-14 12:39:39,039 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32930.05 MB 2025-02-14 12:39:39,039 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 12:39:39,039 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38824.57 MB 2025-02-14 12:39:39,039 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39241.91 MB 2025-02-14 12:39:39,039 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 12:39:39,039 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33637.84 MB 2025-02-14 12:39:39,058 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 12:39:39,058 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 12:39:39,058 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:39:39,058 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:39:39,058 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33342.94 MB 2025-02-14 12:39:39,058 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33571.95 MB 2025-02-14 12:39:39,058 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.01 MB 2025-02-14 12:39:39,058 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39241.91 MB 2025-02-14 12:39:39,058 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39241.91 MB 2025-02-14 12:39:39,058 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:39:39,058 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33773.39 MB 2025-02-14 12:39:39,059 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 12:39:39,059 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 12:39:39,059 - resource_logging.py:150 - __exit__ - DEBUG - Time: 32.84 seconds 2025-02-14 12:39:39,059 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:39:39,059 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19797.50 MB 2025-02-14 12:39:39,059 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33772.88 MB 2025-02-14 12:39:39,059 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13975.38 MB 2025-02-14 12:39:39,059 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46051.36 MB 2025-02-14 12:39:39,059 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39241.91 MB 2025-02-14 12:39:39,059 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6809.45 MB 2025-02-14 12:39:39,059 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33773.39 MB 2025-02-14 12:39:39,328 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 12:39:39,328 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 12:39:39,328 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 12:39:39,328 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:39:39,328 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33772.88 MB 2025-02-14 12:39:39,328 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24799.61 MB 2025-02-14 12:39:39,328 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8973.27 MB 2025-02-14 12:39:39,328 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39241.91 MB 2025-02-14 12:39:39,329 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39241.91 MB 2025-02-14 12:39:39,329 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:39:39,329 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36282.70 MB 2025-02-14 12:39:39,347 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8156, cut from 8158 2025-02-14 12:39:39,347 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 12:39:39,354 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 12:39:39,354 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 12:39:39,354 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:39:39,354 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:39:39,354 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24799.61 MB 2025-02-14 12:39:39,354 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33232.90 MB 2025-02-14 12:39:39,354 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8433.30 MB 2025-02-14 12:39:39,354 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39241.91 MB 2025-02-14 12:39:39,354 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47626.32 MB 2025-02-14 12:39:39,354 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8384.41 MB 2025-02-14 12:39:39,354 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33232.90 MB 2025-02-14 12:39:39,504 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7948] 2025-02-14 12:39:39,506 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:39:39,506 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 12:39:39,507 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:39:39,507 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 12:39:39,511 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 12:39:39,512 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:39:39,512 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 12:39:39,512 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 12:40:27,209 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:40:27,209 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 12:40:27,214 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 12:40:27,218 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:40:27,218 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 107, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 12:40:27,219 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:40:27,219 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 107, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 12:40:28,892 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 12:40:28,892 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 12:40:28,892 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.67 seconds 2025-02-14 12:40:28,892 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:40:28,892 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13714.30 MB 2025-02-14 12:40:28,892 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14092.97 MB 2025-02-14 12:40:28,892 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 378.67 MB 2025-02-14 12:40:28,892 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56010.74 MB 2025-02-14 12:40:28,892 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 16909.34 MB 2025-02-14 12:40:28,892 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -39101.40 MB 2025-02-14 12:40:28,892 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22959.18 MB 2025-02-14 12:40:28,895 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 12:40:28,895 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 12:40:28,895 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 12:40:28,895 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:40:28,895 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14092.97 MB 2025-02-14 12:40:28,895 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14276.43 MB 2025-02-14 12:40:28,895 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 183.46 MB 2025-02-14 12:40:28,895 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 16909.34 MB 2025-02-14 12:40:28,895 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 16909.34 MB 2025-02-14 12:40:28,895 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:40:28,895 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14844.49 MB 2025-02-14 12:40:29,415 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 12:40:29,415 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 12:40:29,415 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.52 seconds 2025-02-14 12:40:29,415 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:40:29,415 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14276.43 MB 2025-02-14 12:40:29,415 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14418.43 MB 2025-02-14 12:40:29,415 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 142.00 MB 2025-02-14 12:40:29,415 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 16909.34 MB 2025-02-14 12:40:29,415 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 16909.34 MB 2025-02-14 12:40:29,415 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:40:29,415 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18362.97 MB 2025-02-14 12:40:29,422 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 12:40:29,422 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 12:40:29,422 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 12:40:29,422 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:40:29,422 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14418.36 MB 2025-02-14 12:40:29,422 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14923.69 MB 2025-02-14 12:40:29,422 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 505.33 MB 2025-02-14 12:40:29,422 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 16909.34 MB 2025-02-14 12:40:29,422 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 16909.34 MB 2025-02-14 12:40:29,422 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:40:29,422 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15302.86 MB 2025-02-14 12:40:29,528 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 12:40:29,528 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 12:40:29,528 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 12:40:29,528 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:40:29,528 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14923.69 MB 2025-02-14 12:40:29,528 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15537.46 MB 2025-02-14 12:40:29,528 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 613.77 MB 2025-02-14 12:40:29,528 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 16909.34 MB 2025-02-14 12:40:29,528 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17670.60 MB 2025-02-14 12:40:29,528 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 761.27 MB 2025-02-14 12:40:29,528 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17006.48 MB 2025-02-14 12:40:29,528 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 12:40:29,528 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 12:40:29,529 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 12:40:29,529 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:40:29,529 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14418.36 MB 2025-02-14 12:40:29,529 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15537.46 MB 2025-02-14 12:40:29,529 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1119.10 MB 2025-02-14 12:40:29,529 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 16909.34 MB 2025-02-14 12:40:29,529 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17670.60 MB 2025-02-14 12:40:29,529 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 761.27 MB 2025-02-14 12:40:29,529 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17006.48 MB 2025-02-14 12:40:29,586 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 12:40:29,586 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 12:40:29,586 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-14 12:40:29,586 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:40:29,586 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16130.79 MB 2025-02-14 12:40:29,586 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16388.56 MB 2025-02-14 12:40:29,586 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 257.77 MB 2025-02-14 12:40:29,586 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17670.60 MB 2025-02-14 12:40:29,586 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17834.18 MB 2025-02-14 12:40:29,586 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 163.58 MB 2025-02-14 12:40:29,586 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16577.89 MB 2025-02-14 12:40:29,592 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 12:40:29,592 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 12:40:29,592 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 12:40:29,592 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:40:29,592 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16551.61 MB 2025-02-14 12:40:29,592 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16780.16 MB 2025-02-14 12:40:29,592 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.55 MB 2025-02-14 12:40:29,592 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17834.18 MB 2025-02-14 12:40:29,592 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17834.18 MB 2025-02-14 12:40:29,592 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:40:29,592 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16780.16 MB 2025-02-14 12:40:29,593 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 12:40:29,594 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 12:40:29,594 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.37 seconds 2025-02-14 12:40:29,594 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:40:29,594 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13341.50 MB 2025-02-14 12:40:29,594 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16981.06 MB 2025-02-14 12:40:29,594 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3639.55 MB 2025-02-14 12:40:29,594 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56010.74 MB 2025-02-14 12:40:29,594 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17834.18 MB 2025-02-14 12:40:29,594 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -38176.56 MB 2025-02-14 12:40:29,594 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16981.06 MB 2025-02-14 12:40:29,860 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 12:40:29,860 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 12:40:29,860 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 12:40:29,860 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:40:29,860 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14047.97 MB 2025-02-14 12:40:29,860 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17059.42 MB 2025-02-14 12:40:29,860 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3011.45 MB 2025-02-14 12:40:29,860 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17834.18 MB 2025-02-14 12:40:29,860 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18773.70 MB 2025-02-14 12:40:29,860 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 939.52 MB 2025-02-14 12:40:29,860 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17360.90 MB 2025-02-14 12:40:29,878 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8155, cut from 8157 2025-02-14 12:40:29,878 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 12:40:29,884 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 12:40:29,884 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 12:40:29,884 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:40:29,885 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:40:29,885 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17059.42 MB 2025-02-14 12:40:29,885 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25490.88 MB 2025-02-14 12:40:29,885 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8431.46 MB 2025-02-14 12:40:29,885 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18773.70 MB 2025-02-14 12:40:29,885 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29255.27 MB 2025-02-14 12:40:29,885 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10481.57 MB 2025-02-14 12:40:29,885 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25490.88 MB 2025-02-14 12:40:30,037 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7947] 2025-02-14 12:40:30,038 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:40:30,038 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 12:40:30,039 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:40:30,039 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 12:40:30,044 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 12:40:30,045 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:40:30,045 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 12:40:30,045 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 12:41:37,156 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:41:37,156 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 12:41:37,167 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 12:41:37,174 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:41:37,174 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1004, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 12:41:37,176 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:41:37,176 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1004, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 12:41:52,583 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 12:41:52,583 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 12:41:52,583 - resource_logging.py:150 - __exit__ - DEBUG - Time: 15.40 seconds 2025-02-14 12:41:52,584 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:41:52,584 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25139.40 MB 2025-02-14 12:41:52,584 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28692.50 MB 2025-02-14 12:41:52,584 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3553.10 MB 2025-02-14 12:41:52,584 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37639.68 MB 2025-02-14 12:41:52,584 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35395.73 MB 2025-02-14 12:41:52,584 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2243.95 MB 2025-02-14 12:41:52,584 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37555.17 MB 2025-02-14 12:41:52,652 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 12:41:52,652 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 12:41:52,652 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 12:41:52,652 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:41:52,652 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28692.50 MB 2025-02-14 12:41:52,652 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20997.33 MB 2025-02-14 12:41:52,652 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7695.17 MB 2025-02-14 12:41:52,652 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35395.73 MB 2025-02-14 12:41:52,652 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40365.98 MB 2025-02-14 12:41:52,652 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4970.25 MB 2025-02-14 12:41:52,652 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34836.21 MB 2025-02-14 12:41:54,566 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 12:41:54,567 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 12:41:54,567 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 12:41:54,567 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:41:54,567 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20997.33 MB 2025-02-14 12:41:54,567 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21528.18 MB 2025-02-14 12:41:54,567 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 12:41:54,567 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40365.98 MB 2025-02-14 12:41:54,567 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28806.48 MB 2025-02-14 12:41:54,567 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11559.50 MB 2025-02-14 12:41:54,567 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25507.51 MB 2025-02-14 12:41:54,580 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 12:41:54,580 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 12:41:54,580 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:41:54,580 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:41:54,580 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21528.18 MB 2025-02-14 12:41:54,580 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23417.71 MB 2025-02-14 12:41:54,580 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 12:41:54,580 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28806.48 MB 2025-02-14 12:41:54,580 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28806.48 MB 2025-02-14 12:41:54,580 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:41:54,580 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24835.14 MB 2025-02-14 12:41:54,798 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 12:41:54,798 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 12:41:54,798 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 12:41:54,798 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:41:54,798 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23417.71 MB 2025-02-14 12:41:54,798 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25659.57 MB 2025-02-14 12:41:54,798 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 12:41:54,798 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28806.48 MB 2025-02-14 12:41:54,798 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33996.93 MB 2025-02-14 12:41:54,798 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 12:41:54,798 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31203.85 MB 2025-02-14 12:41:54,798 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 12:41:54,798 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 12:41:54,798 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 12:41:54,798 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:41:54,798 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21528.18 MB 2025-02-14 12:41:54,798 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25659.57 MB 2025-02-14 12:41:54,799 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 12:41:54,799 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28806.48 MB 2025-02-14 12:41:54,799 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33996.93 MB 2025-02-14 12:41:54,799 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 12:41:54,799 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31203.85 MB 2025-02-14 12:41:54,973 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 12:41:54,973 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 12:41:54,973 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 12:41:54,973 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:41:54,973 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27193.11 MB 2025-02-14 12:41:54,973 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27960.11 MB 2025-02-14 12:41:54,973 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 12:41:54,973 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33996.93 MB 2025-02-14 12:41:54,973 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34412.17 MB 2025-02-14 12:41:54,973 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 12:41:54,973 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28667.90 MB 2025-02-14 12:41:54,993 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 12:41:54,993 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 12:41:54,993 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:41:54,993 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:41:54,993 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28373.00 MB 2025-02-14 12:41:54,993 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28601.70 MB 2025-02-14 12:41:54,993 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.70 MB 2025-02-14 12:41:54,993 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34412.17 MB 2025-02-14 12:41:54,993 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34412.17 MB 2025-02-14 12:41:54,993 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:41:54,993 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28806.26 MB 2025-02-14 12:41:54,994 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 12:41:54,994 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 12:41:54,994 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.81 seconds 2025-02-14 12:41:54,994 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:41:54,994 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21641.39 MB 2025-02-14 12:41:54,994 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28802.73 MB 2025-02-14 12:41:54,994 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7161.34 MB 2025-02-14 12:41:54,994 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37639.68 MB 2025-02-14 12:41:54,994 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34412.17 MB 2025-02-14 12:41:54,994 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3227.52 MB 2025-02-14 12:41:54,994 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28806.26 MB 2025-02-14 12:41:55,263 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 12:41:55,263 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 12:41:55,263 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 12:41:55,263 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:41:55,263 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28802.73 MB 2025-02-14 12:41:55,263 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21470.35 MB 2025-02-14 12:41:55,263 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7332.38 MB 2025-02-14 12:41:55,263 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34412.17 MB 2025-02-14 12:41:55,263 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34412.17 MB 2025-02-14 12:41:55,263 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:41:55,263 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31313.78 MB 2025-02-14 12:41:55,281 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8160, cut from 8162 2025-02-14 12:41:55,281 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 12:41:55,287 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 12:41:55,287 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 12:41:55,287 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:41:55,287 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:41:55,287 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21470.35 MB 2025-02-14 12:41:55,287 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29907.82 MB 2025-02-14 12:41:55,287 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8437.47 MB 2025-02-14 12:41:55,287 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34412.17 MB 2025-02-14 12:41:55,287 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42800.78 MB 2025-02-14 12:41:55,287 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8388.61 MB 2025-02-14 12:41:55,287 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29907.82 MB 2025-02-14 12:41:55,456 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7952] 2025-02-14 12:41:55,457 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:41:55,457 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 12:41:55,458 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:41:55,458 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 12:41:55,463 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 12:41:55,464 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:41:55,464 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 12:41:55,464 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 12:42:12,260 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:42:12,261 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 12:42:12,265 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 12:42:12,268 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:42:12,268 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1561, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 12:42:12,269 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:42:12,269 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1561, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 12:42:36,522 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 12:42:36,523 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 12:42:36,523 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.25 seconds 2025-02-14 12:42:36,523 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:42:36,523 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23846.00 MB 2025-02-14 12:42:36,523 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29370.30 MB 2025-02-14 12:42:36,523 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5524.29 MB 2025-02-14 12:42:36,523 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51189.38 MB 2025-02-14 12:42:36,523 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39210.45 MB 2025-02-14 12:42:36,523 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11978.93 MB 2025-02-14 12:42:36,523 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38301.02 MB 2025-02-14 12:42:36,615 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 12:42:36,615 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 12:42:36,615 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 12:42:36,616 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:42:36,616 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29370.30 MB 2025-02-14 12:42:36,616 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23893.00 MB 2025-02-14 12:42:36,616 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5477.29 MB 2025-02-14 12:42:36,616 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39210.45 MB 2025-02-14 12:42:36,616 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50556.04 MB 2025-02-14 12:42:36,616 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 11345.59 MB 2025-02-14 12:42:36,616 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45172.52 MB 2025-02-14 12:42:38,557 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 12:42:38,557 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 12:42:38,557 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-14 12:42:38,557 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:42:38,557 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23893.00 MB 2025-02-14 12:42:38,557 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24423.84 MB 2025-02-14 12:42:38,557 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 12:42:38,557 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50556.04 MB 2025-02-14 12:42:38,557 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29492.25 MB 2025-02-14 12:42:38,557 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21063.79 MB 2025-02-14 12:42:38,557 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28403.18 MB 2025-02-14 12:42:38,573 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 12:42:38,573 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 12:42:38,573 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:42:38,573 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:42:38,573 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24423.84 MB 2025-02-14 12:42:38,573 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26313.38 MB 2025-02-14 12:42:38,573 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 12:42:38,573 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29492.25 MB 2025-02-14 12:42:38,573 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30435.97 MB 2025-02-14 12:42:38,573 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 12:42:38,573 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27730.81 MB 2025-02-14 12:42:38,788 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 12:42:38,788 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 12:42:38,788 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 12:42:38,788 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:42:38,788 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26313.38 MB 2025-02-14 12:42:38,788 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28555.23 MB 2025-02-14 12:42:38,789 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 12:42:38,789 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30435.97 MB 2025-02-14 12:42:38,789 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36098.28 MB 2025-02-14 12:42:38,789 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 12:42:38,789 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34099.52 MB 2025-02-14 12:42:38,789 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 12:42:38,789 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 12:42:38,789 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 12:42:38,789 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:42:38,789 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24423.84 MB 2025-02-14 12:42:38,789 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28555.23 MB 2025-02-14 12:42:38,789 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 12:42:38,789 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29492.25 MB 2025-02-14 12:42:38,789 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36098.28 MB 2025-02-14 12:42:38,789 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 12:42:38,789 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34099.52 MB 2025-02-14 12:42:38,966 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 12:42:38,966 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 12:42:38,966 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 12:42:38,966 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:42:38,966 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30088.78 MB 2025-02-14 12:42:38,966 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30855.78 MB 2025-02-14 12:42:38,967 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 12:42:38,967 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36098.28 MB 2025-02-14 12:42:38,967 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36511.42 MB 2025-02-14 12:42:38,967 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 12:42:38,967 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31563.57 MB 2025-02-14 12:42:38,986 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 12:42:38,986 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 12:42:38,986 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:42:38,986 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:42:38,986 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31268.67 MB 2025-02-14 12:42:38,986 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31497.16 MB 2025-02-14 12:42:38,986 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.49 MB 2025-02-14 12:42:38,986 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36511.42 MB 2025-02-14 12:42:38,986 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36511.42 MB 2025-02-14 12:42:38,986 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:42:38,986 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31734.17 MB 2025-02-14 12:42:38,988 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 12:42:38,988 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 12:42:38,988 - resource_logging.py:150 - __exit__ - DEBUG - Time: 26.72 seconds 2025-02-14 12:42:38,988 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:42:38,988 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18407.36 MB 2025-02-14 12:42:38,988 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31698.16 MB 2025-02-14 12:42:38,988 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13290.81 MB 2025-02-14 12:42:38,988 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51189.38 MB 2025-02-14 12:42:38,988 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36511.42 MB 2025-02-14 12:42:38,988 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14677.97 MB 2025-02-14 12:42:38,988 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31734.17 MB 2025-02-14 12:42:39,258 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 12:42:39,258 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 12:42:39,258 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 12:42:39,258 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:42:39,258 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31698.16 MB 2025-02-14 12:42:39,258 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23410.60 MB 2025-02-14 12:42:39,258 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8287.56 MB 2025-02-14 12:42:39,258 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36511.42 MB 2025-02-14 12:42:39,258 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36511.42 MB 2025-02-14 12:42:39,258 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:42:39,258 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34208.91 MB 2025-02-14 12:42:39,276 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8159, cut from 8161 2025-02-14 12:42:39,276 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 12:42:39,282 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 12:42:39,282 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 12:42:39,282 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:42:39,282 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:42:39,282 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23410.60 MB 2025-02-14 12:42:39,282 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31846.20 MB 2025-02-14 12:42:39,282 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8435.59 MB 2025-02-14 12:42:39,282 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36511.42 MB 2025-02-14 12:42:39,282 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44900.02 MB 2025-02-14 12:42:39,283 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8388.61 MB 2025-02-14 12:42:39,283 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31846.20 MB 2025-02-14 12:42:39,450 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7951] 2025-02-14 12:42:39,451 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:42:39,451 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 12:42:39,452 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:42:39,452 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 12:42:39,457 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 12:42:39,458 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:42:39,458 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 12:42:39,458 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 12:43:32,598 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:43:32,598 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 12:43:32,605 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 12:43:32,611 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:43:32,611 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 354, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 12:43:32,613 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:43:32,613 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 354, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 12:43:38,209 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 12:43:38,209 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 12:43:38,209 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.59 seconds 2025-02-14 12:43:38,209 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:43:38,209 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15435.44 MB 2025-02-14 12:43:38,209 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16688.22 MB 2025-02-14 12:43:38,209 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1252.79 MB 2025-02-14 12:43:38,209 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53288.63 MB 2025-02-14 12:43:38,209 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20052.97 MB 2025-02-14 12:43:38,209 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33235.66 MB 2025-02-14 12:43:38,209 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25587.09 MB 2025-02-14 12:43:38,232 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 12:43:38,232 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 12:43:38,232 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:43:38,232 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:43:38,232 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16688.22 MB 2025-02-14 12:43:38,232 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17289.58 MB 2025-02-14 12:43:38,232 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 601.36 MB 2025-02-14 12:43:38,232 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20052.97 MB 2025-02-14 12:43:38,232 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23764.93 MB 2025-02-14 12:43:38,232 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3711.96 MB 2025-02-14 12:43:38,232 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21656.50 MB 2025-02-14 12:43:39,927 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 12:43:39,927 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 12:43:39,927 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.69 seconds 2025-02-14 12:43:39,927 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:43:39,927 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17289.58 MB 2025-02-14 12:43:39,927 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17758.05 MB 2025-02-14 12:43:39,927 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 468.47 MB 2025-02-14 12:43:39,927 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23764.93 MB 2025-02-14 12:43:39,927 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20654.85 MB 2025-02-14 12:43:39,927 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3110.08 MB 2025-02-14 12:43:39,927 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21715.86 MB 2025-02-14 12:43:39,940 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 12:43:39,940 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 12:43:39,940 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:43:39,940 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:43:39,940 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17758.05 MB 2025-02-14 12:43:39,940 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19425.55 MB 2025-02-14 12:43:39,940 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1667.50 MB 2025-02-14 12:43:39,940 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20654.85 MB 2025-02-14 12:43:39,940 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23158.85 MB 2025-02-14 12:43:39,940 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2504.00 MB 2025-02-14 12:43:39,940 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20676.43 MB 2025-02-14 12:43:40,125 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 12:43:40,126 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 12:43:40,126 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-14 12:43:40,126 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:43:40,126 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19425.55 MB 2025-02-14 12:43:40,126 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21403.99 MB 2025-02-14 12:43:40,126 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1978.45 MB 2025-02-14 12:43:40,126 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23158.85 MB 2025-02-14 12:43:40,126 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28584.18 MB 2025-02-14 12:43:40,126 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5425.33 MB 2025-02-14 12:43:40,126 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26297.73 MB 2025-02-14 12:43:40,126 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 12:43:40,126 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 12:43:40,126 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 12:43:40,126 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:43:40,126 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17758.05 MB 2025-02-14 12:43:40,126 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21403.99 MB 2025-02-14 12:43:40,126 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3645.94 MB 2025-02-14 12:43:40,126 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20654.85 MB 2025-02-14 12:43:40,126 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28584.18 MB 2025-02-14 12:43:40,126 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7929.33 MB 2025-02-14 12:43:40,126 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26297.73 MB 2025-02-14 12:43:40,268 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 12:43:40,268 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 12:43:40,268 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-14 12:43:40,268 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:43:40,268 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22757.34 MB 2025-02-14 12:43:40,268 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23435.14 MB 2025-02-14 12:43:40,269 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 677.80 MB 2025-02-14 12:43:40,269 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28584.18 MB 2025-02-14 12:43:40,269 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28946.99 MB 2025-02-14 12:43:40,269 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 362.81 MB 2025-02-14 12:43:40,269 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24059.76 MB 2025-02-14 12:43:40,285 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 12:43:40,285 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 12:43:40,285 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:43:40,285 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:43:40,286 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23799.52 MB 2025-02-14 12:43:40,286 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24030.29 MB 2025-02-14 12:43:40,286 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 230.77 MB 2025-02-14 12:43:40,286 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28946.99 MB 2025-02-14 12:43:40,286 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28946.99 MB 2025-02-14 12:43:40,286 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:43:40,286 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24195.35 MB 2025-02-14 12:43:40,287 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 12:43:40,287 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 12:43:40,287 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.67 seconds 2025-02-14 12:43:40,287 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:43:40,287 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14202.07 MB 2025-02-14 12:43:40,287 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24231.36 MB 2025-02-14 12:43:40,287 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 10029.29 MB 2025-02-14 12:43:40,287 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53288.63 MB 2025-02-14 12:43:40,287 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28946.99 MB 2025-02-14 12:43:40,287 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24341.64 MB 2025-02-14 12:43:40,287 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24231.36 MB 2025-02-14 12:43:40,554 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 12:43:40,554 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 12:43:40,554 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 12:43:40,554 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:43:40,554 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24231.36 MB 2025-02-14 12:43:40,554 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18985.31 MB 2025-02-14 12:43:40,554 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5246.05 MB 2025-02-14 12:43:40,554 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28946.99 MB 2025-02-14 12:43:40,554 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28946.99 MB 2025-02-14 12:43:40,554 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:43:40,554 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27446.29 MB 2025-02-14 12:43:40,572 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 12:43:40,573 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 12:43:40,579 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 12:43:40,579 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 12:43:40,579 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:43:40,579 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:43:40,579 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18985.31 MB 2025-02-14 12:43:40,579 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27424.33 MB 2025-02-14 12:43:40,579 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 12:43:40,579 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28946.99 MB 2025-02-14 12:43:40,579 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39436.94 MB 2025-02-14 12:43:40,579 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 12:43:40,579 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27424.33 MB 2025-02-14 12:43:40,729 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 12:43:40,731 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:43:40,731 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 12:43:40,732 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:43:40,732 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 12:43:40,736 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 12:43:40,737 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:43:40,737 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 12:43:40,737 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 12:43:50,311 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:43:50,312 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 12:43:50,316 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 12:43:50,320 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:43:50,320 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1132, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 12:43:50,321 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:43:50,321 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1132, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 12:44:07,883 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 12:44:07,883 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 12:44:07,883 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.56 seconds 2025-02-14 12:44:07,883 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:44:07,883 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20856.66 MB 2025-02-14 12:44:07,883 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24862.75 MB 2025-02-14 12:44:07,883 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4006.08 MB 2025-02-14 12:44:07,883 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52021.95 MB 2025-02-14 12:44:07,883 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31402.75 MB 2025-02-14 12:44:07,883 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20619.20 MB 2025-02-14 12:44:07,883 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33725.42 MB 2025-02-14 12:44:07,952 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 12:44:07,952 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 12:44:07,952 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 12:44:07,952 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:44:07,952 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24862.75 MB 2025-02-14 12:44:07,952 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21662.77 MB 2025-02-14 12:44:07,952 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3199.98 MB 2025-02-14 12:44:07,952 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31402.75 MB 2025-02-14 12:44:07,952 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42482.01 MB 2025-02-14 12:44:07,952 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 11079.25 MB 2025-02-14 12:44:07,952 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37081.09 MB 2025-02-14 12:44:09,873 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 12:44:09,873 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 12:44:09,873 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 12:44:09,873 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:44:09,873 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21662.77 MB 2025-02-14 12:44:09,873 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22193.61 MB 2025-02-14 12:44:09,873 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 12:44:09,873 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42482.01 MB 2025-02-14 12:44:09,873 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28810.67 MB 2025-02-14 12:44:09,873 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13671.33 MB 2025-02-14 12:44:09,873 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26172.94 MB 2025-02-14 12:44:09,887 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 12:44:09,887 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 12:44:09,887 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:44:09,887 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:44:09,887 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22193.61 MB 2025-02-14 12:44:09,887 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24083.14 MB 2025-02-14 12:44:09,887 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 12:44:09,887 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28810.67 MB 2025-02-14 12:44:09,887 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28810.67 MB 2025-02-14 12:44:09,887 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:44:09,887 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25500.57 MB 2025-02-14 12:44:10,102 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 12:44:10,102 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 12:44:10,102 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 12:44:10,102 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:44:10,102 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24083.14 MB 2025-02-14 12:44:10,102 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26325.00 MB 2025-02-14 12:44:10,102 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 12:44:10,102 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28810.67 MB 2025-02-14 12:44:10,102 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34472.98 MB 2025-02-14 12:44:10,102 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 12:44:10,102 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31869.28 MB 2025-02-14 12:44:10,102 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 12:44:10,103 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 12:44:10,103 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 12:44:10,103 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:44:10,103 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22193.61 MB 2025-02-14 12:44:10,103 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26325.00 MB 2025-02-14 12:44:10,103 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 12:44:10,103 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28810.67 MB 2025-02-14 12:44:10,103 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34472.98 MB 2025-02-14 12:44:10,103 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 12:44:10,103 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31869.28 MB 2025-02-14 12:44:10,266 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 12:44:10,266 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 12:44:10,266 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 12:44:10,266 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:44:10,266 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27858.54 MB 2025-02-14 12:44:10,266 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28625.54 MB 2025-02-14 12:44:10,266 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 12:44:10,266 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34472.98 MB 2025-02-14 12:44:10,266 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34890.32 MB 2025-02-14 12:44:10,266 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 12:44:10,266 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29333.33 MB 2025-02-14 12:44:10,284 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 12:44:10,284 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 12:44:10,284 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:44:10,284 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:44:10,284 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29038.43 MB 2025-02-14 12:44:10,284 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29267.37 MB 2025-02-14 12:44:10,284 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.94 MB 2025-02-14 12:44:10,284 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34890.32 MB 2025-02-14 12:44:10,284 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34890.32 MB 2025-02-14 12:44:10,284 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:44:10,284 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29496.41 MB 2025-02-14 12:44:10,285 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 12:44:10,286 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 12:44:10,286 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.96 seconds 2025-02-14 12:44:10,286 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:44:10,286 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16912.69 MB 2025-02-14 12:44:10,286 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29467.97 MB 2025-02-14 12:44:10,286 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12555.29 MB 2025-02-14 12:44:10,286 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52021.95 MB 2025-02-14 12:44:10,286 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34890.32 MB 2025-02-14 12:44:10,286 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17131.63 MB 2025-02-14 12:44:10,286 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29496.41 MB 2025-02-14 12:44:10,556 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 12:44:10,556 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 12:44:10,556 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 12:44:10,556 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:44:10,556 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29467.97 MB 2025-02-14 12:44:10,556 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21909.84 MB 2025-02-14 12:44:10,556 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7558.14 MB 2025-02-14 12:44:10,556 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34890.32 MB 2025-02-14 12:44:10,556 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34890.32 MB 2025-02-14 12:44:10,556 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:44:10,556 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31973.80 MB 2025-02-14 12:44:10,574 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8143, cut from 8145 2025-02-14 12:44:10,574 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 12:44:10,580 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 12:44:10,580 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 12:44:10,580 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:44:10,580 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:44:10,580 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21909.84 MB 2025-02-14 12:44:10,580 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30328.92 MB 2025-02-14 12:44:10,580 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8419.08 MB 2025-02-14 12:44:10,580 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34890.32 MB 2025-02-14 12:44:10,580 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43262.15 MB 2025-02-14 12:44:10,580 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8371.83 MB 2025-02-14 12:44:10,580 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30328.92 MB 2025-02-14 12:44:10,731 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7935] 2025-02-14 12:44:10,732 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:44:10,732 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 12:44:10,733 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:44:10,733 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 12:44:10,738 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 12:44:10,739 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:44:10,739 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 12:44:10,739 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 12:45:35,487 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:45:35,488 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 12:45:35,495 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 12:45:35,502 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:45:35,502 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 186, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 12:45:35,504 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:45:35,504 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 186, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 12:45:38,502 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 12:45:38,502 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 12:45:38,502 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.99 seconds 2025-02-14 12:45:38,502 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:45:38,502 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14264.78 MB 2025-02-14 12:45:38,502 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14923.03 MB 2025-02-14 12:45:38,502 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 658.24 MB 2025-02-14 12:45:38,502 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51633.98 MB 2025-02-14 12:45:38,503 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17855.15 MB 2025-02-14 12:45:38,503 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33778.83 MB 2025-02-14 12:45:38,503 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23736.96 MB 2025-02-14 12:45:38,523 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 12:45:38,523 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 12:45:38,524 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:45:38,524 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:45:38,524 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14923.03 MB 2025-02-14 12:45:38,524 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15242.21 MB 2025-02-14 12:45:38,524 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 319.18 MB 2025-02-14 12:45:38,524 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17855.15 MB 2025-02-14 12:45:38,524 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19172.16 MB 2025-02-14 12:45:38,524 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1317.01 MB 2025-02-14 12:45:38,524 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17571.57 MB 2025-02-14 12:45:39,498 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 12:45:39,498 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 12:45:39,498 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.97 seconds 2025-02-14 12:45:39,498 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:45:39,498 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15242.21 MB 2025-02-14 12:45:39,498 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15489.05 MB 2025-02-14 12:45:39,498 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 246.84 MB 2025-02-14 12:45:39,498 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19172.16 MB 2025-02-14 12:45:39,498 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18916.31 MB 2025-02-14 12:45:39,498 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -255.85 MB 2025-02-14 12:45:39,498 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19413.68 MB 2025-02-14 12:45:39,511 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 12:45:39,511 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 12:45:39,511 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:45:39,511 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:45:39,511 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15488.98 MB 2025-02-14 12:45:39,511 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16367.40 MB 2025-02-14 12:45:39,511 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 878.42 MB 2025-02-14 12:45:39,511 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18916.31 MB 2025-02-14 12:45:39,511 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18916.31 MB 2025-02-14 12:45:39,511 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:45:39,511 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17026.51 MB 2025-02-14 12:45:39,640 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 12:45:39,640 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 12:45:39,640 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 12:45:39,640 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:45:39,640 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16367.40 MB 2025-02-14 12:45:39,640 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17409.90 MB 2025-02-14 12:45:39,640 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1042.50 MB 2025-02-14 12:45:39,640 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18916.31 MB 2025-02-14 12:45:39,640 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21558.72 MB 2025-02-14 12:45:39,641 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2642.41 MB 2025-02-14 12:45:39,641 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19987.96 MB 2025-02-14 12:45:39,642 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 12:45:39,642 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 12:45:39,642 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-14 12:45:39,642 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:45:39,642 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15488.98 MB 2025-02-14 12:45:39,642 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17409.90 MB 2025-02-14 12:45:39,642 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1920.92 MB 2025-02-14 12:45:39,642 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18916.31 MB 2025-02-14 12:45:39,642 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21558.72 MB 2025-02-14 12:45:39,642 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2642.41 MB 2025-02-14 12:45:39,642 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19987.96 MB 2025-02-14 12:45:39,766 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 12:45:39,766 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 12:45:39,766 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 12:45:39,766 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:45:39,766 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18123.00 MB 2025-02-14 12:45:39,766 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18479.65 MB 2025-02-14 12:45:39,766 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 356.66 MB 2025-02-14 12:45:39,766 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21558.72 MB 2025-02-14 12:45:39,766 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21747.47 MB 2025-02-14 12:45:39,766 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 188.74 MB 2025-02-14 12:45:39,767 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18813.52 MB 2025-02-14 12:45:39,784 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 12:45:39,784 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 12:45:39,784 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:45:39,784 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:45:39,784 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18671.65 MB 2025-02-14 12:45:39,784 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18875.26 MB 2025-02-14 12:45:39,784 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 203.60 MB 2025-02-14 12:45:39,784 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21747.47 MB 2025-02-14 12:45:39,784 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21751.66 MB 2025-02-14 12:45:39,785 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-14 12:45:39,785 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18901.91 MB 2025-02-14 12:45:39,787 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 12:45:39,787 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 12:45:39,787 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.28 seconds 2025-02-14 12:45:39,787 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:45:39,787 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13616.74 MB 2025-02-14 12:45:39,787 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19076.11 MB 2025-02-14 12:45:39,787 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5459.36 MB 2025-02-14 12:45:39,787 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51633.98 MB 2025-02-14 12:45:39,787 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21751.66 MB 2025-02-14 12:45:39,787 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29882.32 MB 2025-02-14 12:45:39,787 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19076.11 MB 2025-02-14 12:45:40,073 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 12:45:40,073 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 12:45:40,073 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.28 seconds 2025-02-14 12:45:40,073 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:45:40,073 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19076.11 MB 2025-02-14 12:45:40,073 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17607.25 MB 2025-02-14 12:45:40,073 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1468.85 MB 2025-02-14 12:45:40,073 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21751.66 MB 2025-02-14 12:45:40,073 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21751.66 MB 2025-02-14 12:45:40,073 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:45:40,073 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19076.12 MB 2025-02-14 12:45:40,093 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8153, cut from 8155 2025-02-14 12:45:40,093 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2,'] 2025-02-14 12:45:40,100 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 12:45:40,101 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 12:45:40,101 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:45:40,101 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:45:40,101 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17607.25 MB 2025-02-14 12:45:40,101 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26037.65 MB 2025-02-14 12:45:40,101 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8430.40 MB 2025-02-14 12:45:40,101 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21751.66 MB 2025-02-14 12:45:40,101 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32226.93 MB 2025-02-14 12:45:40,101 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10475.27 MB 2025-02-14 12:45:40,101 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26037.65 MB 2025-02-14 12:45:40,329 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7945] 2025-02-14 12:45:40,331 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:45:40,331 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 12:45:40,332 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:45:40,332 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 12:45:40,337 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 12:45:40,338 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:45:40,338 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 12:45:40,338 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2,'] 2025-02-14 12:47:25,271 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:47:25,271 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 12:47:25,276 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 12:47:25,280 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:47:25,280 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1890, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 12:47:25,281 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:47:25,281 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1890, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 12:47:54,266 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 12:47:54,266 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 12:47:54,266 - resource_logging.py:150 - __exit__ - DEBUG - Time: 28.98 seconds 2025-02-14 12:47:54,266 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:47:54,266 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26138.53 MB 2025-02-14 12:47:54,266 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32827.13 MB 2025-02-14 12:47:54,266 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6688.60 MB 2025-02-14 12:47:54,266 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40607.15 MB 2025-02-14 12:47:54,266 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40359.69 MB 2025-02-14 12:47:54,266 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -247.46 MB 2025-02-14 12:47:54,266 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41726.00 MB 2025-02-14 12:47:54,390 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 12:47:54,390 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 12:47:54,390 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 12:47:54,390 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:47:54,390 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32827.13 MB 2025-02-14 12:47:54,390 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25603.37 MB 2025-02-14 12:47:54,390 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7223.76 MB 2025-02-14 12:47:54,390 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40359.69 MB 2025-02-14 12:47:54,390 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 62084.09 MB 2025-02-14 12:47:54,390 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 21724.40 MB 2025-02-14 12:47:54,390 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52438.27 MB 2025-02-14 12:47:56,312 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 12:47:56,312 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 12:47:56,312 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 12:47:56,312 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:47:56,312 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25603.37 MB 2025-02-14 12:47:56,312 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26134.21 MB 2025-02-14 12:47:56,312 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 12:47:56,312 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62084.09 MB 2025-02-14 12:47:56,312 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35085.35 MB 2025-02-14 12:47:56,312 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26998.73 MB 2025-02-14 12:47:56,312 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30113.55 MB 2025-02-14 12:47:56,326 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 12:47:56,326 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 12:47:56,326 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:47:56,326 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:47:56,326 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26134.21 MB 2025-02-14 12:47:56,326 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28023.75 MB 2025-02-14 12:47:56,326 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 12:47:56,326 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35085.35 MB 2025-02-14 12:47:56,326 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35085.35 MB 2025-02-14 12:47:56,326 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:47:56,326 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29441.18 MB 2025-02-14 12:47:56,537 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 12:47:56,537 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 12:47:56,537 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 12:47:56,537 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:47:56,537 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28023.75 MB 2025-02-14 12:47:56,537 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30265.60 MB 2025-02-14 12:47:56,537 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 12:47:56,537 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35085.35 MB 2025-02-14 12:47:56,537 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38860.23 MB 2025-02-14 12:47:56,537 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 12:47:56,537 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35809.88 MB 2025-02-14 12:47:56,538 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 12:47:56,538 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 12:47:56,538 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 12:47:56,538 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:47:56,538 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26134.21 MB 2025-02-14 12:47:56,538 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30265.60 MB 2025-02-14 12:47:56,538 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 12:47:56,538 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35085.35 MB 2025-02-14 12:47:56,538 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38860.23 MB 2025-02-14 12:47:56,538 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 12:47:56,538 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35809.88 MB 2025-02-14 12:47:56,722 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 12:47:56,722 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 12:47:56,722 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-14 12:47:56,722 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:47:56,722 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31799.14 MB 2025-02-14 12:47:56,722 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32566.15 MB 2025-02-14 12:47:56,722 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 12:47:56,722 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38860.23 MB 2025-02-14 12:47:56,722 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39275.46 MB 2025-02-14 12:47:56,722 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 12:47:56,722 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33273.94 MB 2025-02-14 12:47:56,743 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 12:47:56,743 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 12:47:56,743 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:47:56,743 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:47:56,743 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32979.04 MB 2025-02-14 12:47:56,743 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33208.14 MB 2025-02-14 12:47:56,743 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.11 MB 2025-02-14 12:47:56,743 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39275.46 MB 2025-02-14 12:47:56,743 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39275.46 MB 2025-02-14 12:47:56,743 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:47:56,743 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33420.45 MB 2025-02-14 12:47:56,744 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 12:47:56,744 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 12:47:56,744 - resource_logging.py:150 - __exit__ - DEBUG - Time: 31.46 seconds 2025-02-14 12:47:56,745 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:47:56,745 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19553.62 MB 2025-02-14 12:47:56,745 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33409.17 MB 2025-02-14 12:47:56,745 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13855.55 MB 2025-02-14 12:47:56,745 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40607.15 MB 2025-02-14 12:47:56,745 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39275.46 MB 2025-02-14 12:47:56,745 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1331.69 MB 2025-02-14 12:47:56,745 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33420.45 MB 2025-02-14 12:47:57,014 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 12:47:57,015 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 12:47:57,015 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 12:47:57,015 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:47:57,015 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33409.17 MB 2025-02-14 12:47:57,015 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24557.24 MB 2025-02-14 12:47:57,015 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8851.92 MB 2025-02-14 12:47:57,015 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39275.46 MB 2025-02-14 12:47:57,015 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39275.46 MB 2025-02-14 12:47:57,015 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:47:57,015 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35920.22 MB 2025-02-14 12:47:57,032 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8160, cut from 8162 2025-02-14 12:47:57,033 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 12:47:57,039 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 12:47:57,039 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 12:47:57,039 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:47:57,039 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:47:57,039 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24557.24 MB 2025-02-14 12:47:57,039 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32994.72 MB 2025-02-14 12:47:57,039 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8437.47 MB 2025-02-14 12:47:57,039 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39275.46 MB 2025-02-14 12:47:57,039 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47664.07 MB 2025-02-14 12:47:57,039 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8388.61 MB 2025-02-14 12:47:57,039 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32994.72 MB 2025-02-14 12:47:57,202 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7952] 2025-02-14 12:47:57,203 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:47:57,203 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 12:47:57,204 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:47:57,204 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 12:47:57,209 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 12:47:57,210 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:47:57,210 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 12:47:57,210 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 12:49:38,215 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:49:38,216 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 12:49:38,220 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 12:49:38,224 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:49:38,224 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2418, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 12:49:38,225 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:49:38,225 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2418, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 12:50:15,790 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 12:50:15,790 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 12:50:15,790 - resource_logging.py:150 - __exit__ - DEBUG - Time: 37.55 seconds 2025-02-14 12:50:15,790 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:50:15,790 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29817.72 MB 2025-02-14 12:50:15,790 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38374.88 MB 2025-02-14 12:50:15,790 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8557.17 MB 2025-02-14 12:50:15,790 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64613.25 MB 2025-02-14 12:50:15,790 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42410.70 MB 2025-02-14 12:50:15,790 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22202.55 MB 2025-02-14 12:50:15,790 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47217.13 MB 2025-02-14 12:50:15,948 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 12:50:15,948 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 12:50:15,948 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 12:50:15,948 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:50:15,948 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38374.88 MB 2025-02-14 12:50:15,948 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28348.28 MB 2025-02-14 12:50:15,948 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10026.61 MB 2025-02-14 12:50:15,948 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42410.70 MB 2025-02-14 12:50:15,948 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 73756.84 MB 2025-02-14 12:50:15,948 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 31346.13 MB 2025-02-14 12:50:15,948 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 62447.18 MB 2025-02-14 12:50:17,913 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 12:50:17,913 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 12:50:17,913 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.96 seconds 2025-02-14 12:50:17,913 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:50:17,913 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28348.28 MB 2025-02-14 12:50:17,913 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28879.12 MB 2025-02-14 12:50:17,913 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 12:50:17,913 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 73756.84 MB 2025-02-14 12:50:17,913 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30987.52 MB 2025-02-14 12:50:17,913 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -42769.32 MB 2025-02-14 12:50:17,913 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32859.49 MB 2025-02-14 12:50:17,927 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 12:50:17,927 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 12:50:17,927 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:50:17,927 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:50:17,927 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28879.12 MB 2025-02-14 12:50:17,927 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30768.26 MB 2025-02-14 12:50:17,927 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.14 MB 2025-02-14 12:50:17,927 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30987.52 MB 2025-02-14 12:50:17,927 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34290.53 MB 2025-02-14 12:50:17,927 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 12:50:17,927 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32185.69 MB 2025-02-14 12:50:18,133 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 12:50:18,133 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 12:50:18,133 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 12:50:18,133 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:50:18,134 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30768.26 MB 2025-02-14 12:50:18,134 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33010.12 MB 2025-02-14 12:50:18,134 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 12:50:18,134 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34290.53 MB 2025-02-14 12:50:18,134 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40896.56 MB 2025-02-14 12:50:18,134 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 12:50:18,134 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38554.40 MB 2025-02-14 12:50:18,134 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 12:50:18,134 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 12:50:18,134 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 12:50:18,134 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:50:18,134 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28879.12 MB 2025-02-14 12:50:18,134 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33010.12 MB 2025-02-14 12:50:18,134 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.00 MB 2025-02-14 12:50:18,134 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30987.52 MB 2025-02-14 12:50:18,134 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40896.56 MB 2025-02-14 12:50:18,134 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9909.04 MB 2025-02-14 12:50:18,134 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38554.40 MB 2025-02-14 12:50:18,298 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 12:50:18,298 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 12:50:18,298 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 12:50:18,298 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:50:18,298 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34543.66 MB 2025-02-14 12:50:18,298 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35310.66 MB 2025-02-14 12:50:18,298 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 12:50:18,298 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40896.56 MB 2025-02-14 12:50:18,298 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41309.70 MB 2025-02-14 12:50:18,298 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 12:50:18,298 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36018.45 MB 2025-02-14 12:50:18,317 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 12:50:18,317 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 12:50:18,317 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:50:18,317 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:50:18,317 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35723.55 MB 2025-02-14 12:50:18,317 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35952.49 MB 2025-02-14 12:50:18,317 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.94 MB 2025-02-14 12:50:18,317 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41309.70 MB 2025-02-14 12:50:18,317 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41309.70 MB 2025-02-14 12:50:18,317 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:50:18,317 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36168.27 MB 2025-02-14 12:50:18,318 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 12:50:18,318 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 12:50:18,318 - resource_logging.py:150 - __exit__ - DEBUG - Time: 40.09 seconds 2025-02-14 12:50:18,318 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:50:18,318 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21393.21 MB 2025-02-14 12:50:18,318 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36153.34 MB 2025-02-14 12:50:18,318 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14760.13 MB 2025-02-14 12:50:18,318 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60332.97 MB 2025-02-14 12:50:18,318 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41309.70 MB 2025-02-14 12:50:18,318 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19023.27 MB 2025-02-14 12:50:18,318 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36168.27 MB 2025-02-14 12:50:18,587 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 12:50:18,587 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 12:50:18,587 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 12:50:18,587 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:50:18,587 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36153.34 MB 2025-02-14 12:50:18,587 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26394.17 MB 2025-02-14 12:50:18,588 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9759.16 MB 2025-02-14 12:50:18,588 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41309.70 MB 2025-02-14 12:50:18,588 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41309.70 MB 2025-02-14 12:50:18,588 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:50:18,588 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38662.24 MB 2025-02-14 12:50:18,605 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8153, cut from 8155 2025-02-14 12:50:18,606 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 12:50:18,612 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 12:50:18,612 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 12:50:18,612 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:50:18,612 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:50:18,612 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26394.17 MB 2025-02-14 12:50:18,612 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34824.57 MB 2025-02-14 12:50:18,612 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8430.40 MB 2025-02-14 12:50:18,612 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41309.70 MB 2025-02-14 12:50:18,612 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45499.81 MB 2025-02-14 12:50:18,612 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4190.11 MB 2025-02-14 12:50:18,612 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34824.57 MB 2025-02-14 12:50:18,763 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7945] 2025-02-14 12:50:18,765 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:50:18,765 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 12:50:18,766 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:50:18,766 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 12:50:18,770 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 12:50:18,771 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:50:18,771 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 12:50:18,772 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 12:51:45,148 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:51:45,148 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 12:51:45,153 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 12:51:45,157 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:51:45,157 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1991, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 12:51:45,158 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:51:45,158 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1991, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 12:52:15,938 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 12:52:15,938 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 12:52:15,938 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.77 seconds 2025-02-14 12:52:15,938 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:52:15,938 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26842.31 MB 2025-02-14 12:52:15,938 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33888.74 MB 2025-02-14 12:52:15,938 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7046.43 MB 2025-02-14 12:52:15,938 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53880.03 MB 2025-02-14 12:52:15,938 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40716.21 MB 2025-02-14 12:52:15,938 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13163.82 MB 2025-02-14 12:52:15,938 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42882.77 MB 2025-02-14 12:52:16,063 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 12:52:16,063 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 12:52:16,063 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 12:52:16,063 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:52:16,063 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33888.74 MB 2025-02-14 12:52:16,063 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26128.44 MB 2025-02-14 12:52:16,063 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7760.31 MB 2025-02-14 12:52:16,063 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40716.21 MB 2025-02-14 12:52:16,063 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 63430.46 MB 2025-02-14 12:52:16,063 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 22714.25 MB 2025-02-14 12:52:16,063 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 53741.10 MB 2025-02-14 12:52:18,010 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 12:52:18,010 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 12:52:18,010 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.95 seconds 2025-02-14 12:52:18,010 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:52:18,010 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26128.44 MB 2025-02-14 12:52:18,010 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26659.28 MB 2025-02-14 12:52:18,010 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 12:52:18,010 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63430.46 MB 2025-02-14 12:52:18,010 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30895.24 MB 2025-02-14 12:52:18,010 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32535.22 MB 2025-02-14 12:52:18,010 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30639.65 MB 2025-02-14 12:52:18,024 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 12:52:18,024 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 12:52:18,024 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:52:18,024 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:52:18,024 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26659.28 MB 2025-02-14 12:52:18,024 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28548.81 MB 2025-02-14 12:52:18,024 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 12:52:18,024 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30895.24 MB 2025-02-14 12:52:18,024 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32782.68 MB 2025-02-14 12:52:18,024 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 12:52:18,024 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29966.24 MB 2025-02-14 12:52:18,239 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 12:52:18,239 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 12:52:18,239 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 12:52:18,239 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:52:18,239 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28548.81 MB 2025-02-14 12:52:18,239 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30790.67 MB 2025-02-14 12:52:18,239 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 12:52:18,239 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32782.68 MB 2025-02-14 12:52:18,239 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38916.85 MB 2025-02-14 12:52:18,239 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 12:52:18,239 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36334.95 MB 2025-02-14 12:52:18,240 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 12:52:18,240 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 12:52:18,240 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 12:52:18,240 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:52:18,240 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26659.28 MB 2025-02-14 12:52:18,240 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30790.67 MB 2025-02-14 12:52:18,240 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 12:52:18,240 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30895.24 MB 2025-02-14 12:52:18,240 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38916.85 MB 2025-02-14 12:52:18,240 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-14 12:52:18,240 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36334.95 MB 2025-02-14 12:52:18,416 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 12:52:18,416 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 12:52:18,416 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 12:52:18,416 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:52:18,416 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32324.21 MB 2025-02-14 12:52:18,416 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33091.21 MB 2025-02-14 12:52:18,416 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 12:52:18,417 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38916.85 MB 2025-02-14 12:52:18,417 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39332.09 MB 2025-02-14 12:52:18,417 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 12:52:18,417 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33799.00 MB 2025-02-14 12:52:18,436 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 12:52:18,436 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 12:52:18,436 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:52:18,436 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:52:18,436 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33504.10 MB 2025-02-14 12:52:18,436 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33732.18 MB 2025-02-14 12:52:18,436 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.08 MB 2025-02-14 12:52:18,437 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39332.09 MB 2025-02-14 12:52:18,437 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39332.09 MB 2025-02-14 12:52:18,437 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:52:18,437 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33956.68 MB 2025-02-14 12:52:18,438 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 12:52:18,438 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 12:52:18,438 - resource_logging.py:150 - __exit__ - DEBUG - Time: 33.28 seconds 2025-02-14 12:52:18,438 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:52:18,438 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19905.51 MB 2025-02-14 12:52:18,438 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33932.17 MB 2025-02-14 12:52:18,438 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14026.66 MB 2025-02-14 12:52:18,438 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53880.03 MB 2025-02-14 12:52:18,438 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39332.09 MB 2025-02-14 12:52:18,438 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14547.94 MB 2025-02-14 12:52:18,438 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33956.68 MB 2025-02-14 12:52:18,709 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 12:52:18,709 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 12:52:18,709 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 12:52:18,709 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:52:18,709 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33932.17 MB 2025-02-14 12:52:18,709 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24893.14 MB 2025-02-14 12:52:18,709 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9039.03 MB 2025-02-14 12:52:18,709 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39332.09 MB 2025-02-14 12:52:18,709 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39332.09 MB 2025-02-14 12:52:18,709 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:52:18,709 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36430.32 MB 2025-02-14 12:52:18,727 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8118, cut from 8120 2025-02-14 12:52:18,727 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 12:52:18,733 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 12:52:18,733 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 12:52:18,733 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:52:18,733 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:52:18,733 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24893.14 MB 2025-02-14 12:52:18,733 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33286.41 MB 2025-02-14 12:52:18,733 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8393.27 MB 2025-02-14 12:52:18,733 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39332.09 MB 2025-02-14 12:52:18,733 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43505.42 MB 2025-02-14 12:52:18,733 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4173.33 MB 2025-02-14 12:52:18,733 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33286.41 MB 2025-02-14 12:52:18,902 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7910] 2025-02-14 12:52:18,903 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:52:18,903 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 12:52:18,904 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:52:18,904 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 12:52:18,909 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 12:52:18,910 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:52:18,910 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 12:52:18,910 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 12:52:54,892 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:52:54,892 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 12:52:54,897 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 12:52:54,901 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:52:54,901 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1860, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 12:52:54,902 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:52:54,902 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1860, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 12:53:23,876 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 12:53:23,876 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 12:53:23,876 - resource_logging.py:150 - __exit__ - DEBUG - Time: 28.96 seconds 2025-02-14 12:53:23,876 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:53:23,876 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25929.48 MB 2025-02-14 12:53:23,876 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32512.44 MB 2025-02-14 12:53:23,876 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6582.96 MB 2025-02-14 12:53:23,876 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51852.08 MB 2025-02-14 12:53:23,876 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40202.40 MB 2025-02-14 12:53:23,876 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11649.68 MB 2025-02-14 12:53:23,876 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41516.96 MB 2025-02-14 12:53:23,986 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 12:53:23,986 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 12:53:23,986 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 12:53:23,986 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:53:23,986 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32512.44 MB 2025-02-14 12:53:23,986 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25447.41 MB 2025-02-14 12:53:23,986 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7065.03 MB 2025-02-14 12:53:23,986 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40202.40 MB 2025-02-14 12:53:23,986 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 59399.73 MB 2025-02-14 12:53:23,986 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 19197.33 MB 2025-02-14 12:53:23,986 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50382.34 MB 2025-02-14 12:53:25,930 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 12:53:25,930 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 12:53:25,930 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-14 12:53:25,930 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:53:25,930 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25447.41 MB 2025-02-14 12:53:25,930 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25978.25 MB 2025-02-14 12:53:25,930 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 12:53:25,930 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59399.73 MB 2025-02-14 12:53:25,930 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30861.69 MB 2025-02-14 12:53:25,930 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28538.04 MB 2025-02-14 12:53:25,930 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29957.59 MB 2025-02-14 12:53:25,947 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 12:53:25,947 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 12:53:25,947 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:53:25,947 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:53:25,947 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25978.25 MB 2025-02-14 12:53:25,947 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27867.79 MB 2025-02-14 12:53:25,947 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 12:53:25,947 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30861.69 MB 2025-02-14 12:53:25,947 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32749.13 MB 2025-02-14 12:53:25,947 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 12:53:25,947 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29285.22 MB 2025-02-14 12:53:26,158 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 12:53:26,158 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 12:53:26,158 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 12:53:26,158 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:53:26,158 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27867.79 MB 2025-02-14 12:53:26,158 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30109.64 MB 2025-02-14 12:53:26,158 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 12:53:26,158 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32749.13 MB 2025-02-14 12:53:26,158 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38411.44 MB 2025-02-14 12:53:26,158 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 12:53:26,158 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35653.92 MB 2025-02-14 12:53:26,159 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 12:53:26,159 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 12:53:26,159 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 12:53:26,159 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:53:26,159 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25978.25 MB 2025-02-14 12:53:26,159 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30109.64 MB 2025-02-14 12:53:26,159 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 12:53:26,159 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30861.69 MB 2025-02-14 12:53:26,159 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38411.44 MB 2025-02-14 12:53:26,159 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-14 12:53:26,159 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35653.92 MB 2025-02-14 12:53:26,331 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 12:53:26,331 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 12:53:26,331 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 12:53:26,331 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:53:26,331 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31643.18 MB 2025-02-14 12:53:26,331 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32410.19 MB 2025-02-14 12:53:26,331 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 12:53:26,331 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38411.44 MB 2025-02-14 12:53:26,331 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38826.67 MB 2025-02-14 12:53:26,331 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 12:53:26,331 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33117.98 MB 2025-02-14 12:53:26,350 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 12:53:26,350 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 12:53:26,350 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:53:26,350 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:53:26,350 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32823.08 MB 2025-02-14 12:53:26,351 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33052.09 MB 2025-02-14 12:53:26,351 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.01 MB 2025-02-14 12:53:26,351 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38826.67 MB 2025-02-14 12:53:26,351 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38826.67 MB 2025-02-14 12:53:26,351 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:53:26,351 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33263.94 MB 2025-02-14 12:53:26,352 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 12:53:26,352 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 12:53:26,352 - resource_logging.py:150 - __exit__ - DEBUG - Time: 31.45 seconds 2025-02-14 12:53:26,352 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:53:26,352 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19449.10 MB 2025-02-14 12:53:26,352 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33253.01 MB 2025-02-14 12:53:26,352 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13803.92 MB 2025-02-14 12:53:26,352 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51852.08 MB 2025-02-14 12:53:26,352 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38826.67 MB 2025-02-14 12:53:26,352 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13025.41 MB 2025-02-14 12:53:26,352 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33263.94 MB 2025-02-14 12:53:26,625 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 12:53:26,625 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 12:53:26,625 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 12:53:26,625 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:53:26,625 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33253.01 MB 2025-02-14 12:53:26,625 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24451.49 MB 2025-02-14 12:53:26,625 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8801.52 MB 2025-02-14 12:53:26,625 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38826.67 MB 2025-02-14 12:53:26,625 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38826.67 MB 2025-02-14 12:53:26,625 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:53:26,625 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35762.83 MB 2025-02-14 12:53:26,643 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8156, cut from 8158 2025-02-14 12:53:26,643 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 12:53:26,649 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 12:53:26,649 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 12:53:26,649 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:53:26,649 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:53:26,649 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24451.49 MB 2025-02-14 12:53:26,649 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32885.55 MB 2025-02-14 12:53:26,649 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8434.05 MB 2025-02-14 12:53:26,649 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38826.67 MB 2025-02-14 12:53:26,649 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47211.09 MB 2025-02-14 12:53:26,649 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8384.41 MB 2025-02-14 12:53:26,649 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32885.55 MB 2025-02-14 12:53:26,811 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7948] 2025-02-14 12:53:26,812 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:53:26,813 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 12:53:26,813 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:53:26,814 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 12:53:26,818 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 12:53:26,819 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:53:26,819 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 12:53:26,819 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 12:54:55,953 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:54:55,953 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 12:54:55,958 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 12:54:55,962 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:54:55,962 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 733, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 12:54:55,963 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:54:55,963 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 733, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 12:55:07,229 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 12:55:07,229 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 12:55:07,229 - resource_logging.py:150 - __exit__ - DEBUG - Time: 11.26 seconds 2025-02-14 12:55:07,229 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:55:07,229 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18076.37 MB 2025-02-14 12:55:07,229 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20670.54 MB 2025-02-14 12:55:07,229 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2594.18 MB 2025-02-14 12:55:07,229 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55595.50 MB 2025-02-14 12:55:07,229 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25090.33 MB 2025-02-14 12:55:07,229 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30505.17 MB 2025-02-14 12:55:07,229 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29586.98 MB 2025-02-14 12:55:07,276 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 12:55:07,276 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 12:55:07,276 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-14 12:55:07,276 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:55:07,276 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20670.54 MB 2025-02-14 12:55:07,276 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19589.54 MB 2025-02-14 12:55:07,276 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1081.01 MB 2025-02-14 12:55:07,276 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25090.33 MB 2025-02-14 12:55:07,276 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32380.03 MB 2025-02-14 12:55:07,276 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7289.70 MB 2025-02-14 12:55:07,276 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29640.99 MB 2025-02-14 12:55:09,185 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 12:55:09,185 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 12:55:09,185 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 12:55:09,185 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:55:09,185 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19589.54 MB 2025-02-14 12:55:09,185 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20120.38 MB 2025-02-14 12:55:09,185 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 12:55:09,185 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32380.03 MB 2025-02-14 12:55:09,185 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24620.56 MB 2025-02-14 12:55:09,185 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7759.46 MB 2025-02-14 12:55:09,185 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24099.71 MB 2025-02-14 12:55:09,199 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 12:55:09,199 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 12:55:09,199 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:55:09,199 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:55:09,199 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20120.38 MB 2025-02-14 12:55:09,199 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22009.91 MB 2025-02-14 12:55:09,199 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 12:55:09,199 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24620.56 MB 2025-02-14 12:55:09,199 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26508.00 MB 2025-02-14 12:55:09,199 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 12:55:09,199 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23427.34 MB 2025-02-14 12:55:09,405 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 12:55:09,405 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 12:55:09,405 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 12:55:09,405 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:55:09,405 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22009.91 MB 2025-02-14 12:55:09,405 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24251.77 MB 2025-02-14 12:55:09,405 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 12:55:09,405 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26508.00 MB 2025-02-14 12:55:09,405 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32170.31 MB 2025-02-14 12:55:09,405 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 12:55:09,405 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29796.05 MB 2025-02-14 12:55:09,406 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 12:55:09,406 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 12:55:09,406 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 12:55:09,406 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:55:09,406 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20120.38 MB 2025-02-14 12:55:09,406 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24251.77 MB 2025-02-14 12:55:09,406 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 12:55:09,406 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24620.56 MB 2025-02-14 12:55:09,406 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32170.31 MB 2025-02-14 12:55:09,406 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-14 12:55:09,406 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29796.05 MB 2025-02-14 12:55:09,571 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 12:55:09,572 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 12:55:09,572 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 12:55:09,572 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:55:09,572 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25785.31 MB 2025-02-14 12:55:09,572 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26552.31 MB 2025-02-14 12:55:09,572 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 12:55:09,572 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32170.31 MB 2025-02-14 12:55:09,572 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32587.64 MB 2025-02-14 12:55:09,572 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 12:55:09,572 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27260.10 MB 2025-02-14 12:55:09,591 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 12:55:09,591 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 12:55:09,591 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:55:09,591 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:55:09,591 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26965.20 MB 2025-02-14 12:55:09,591 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27193.18 MB 2025-02-14 12:55:09,591 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.98 MB 2025-02-14 12:55:09,591 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32587.64 MB 2025-02-14 12:55:09,591 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32587.64 MB 2025-02-14 12:55:09,591 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:55:09,591 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27382.06 MB 2025-02-14 12:55:09,592 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 12:55:09,592 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 12:55:09,592 - resource_logging.py:150 - __exit__ - DEBUG - Time: 13.63 seconds 2025-02-14 12:55:09,592 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:55:09,592 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15522.54 MB 2025-02-14 12:55:09,592 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27394.25 MB 2025-02-14 12:55:09,592 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11871.72 MB 2025-02-14 12:55:09,592 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55595.50 MB 2025-02-14 12:55:09,592 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32587.64 MB 2025-02-14 12:55:09,592 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23007.85 MB 2025-02-14 12:55:09,592 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27394.25 MB 2025-02-14 12:55:09,860 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 12:55:09,860 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 12:55:09,860 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 12:55:09,860 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:55:09,860 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27394.25 MB 2025-02-14 12:55:09,860 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20526.93 MB 2025-02-14 12:55:09,860 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6867.33 MB 2025-02-14 12:55:09,860 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32587.64 MB 2025-02-14 12:55:09,860 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32587.64 MB 2025-02-14 12:55:09,860 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:55:09,860 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29905.92 MB 2025-02-14 12:55:09,882 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 12:55:09,883 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 12:55:09,889 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 12:55:09,889 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 12:55:09,889 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 12:55:09,889 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:55:09,889 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20526.93 MB 2025-02-14 12:55:09,889 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28965.95 MB 2025-02-14 12:55:09,889 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 12:55:09,889 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32587.64 MB 2025-02-14 12:55:09,889 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40978.35 MB 2025-02-14 12:55:09,889 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 12:55:09,889 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28965.95 MB 2025-02-14 12:55:10,040 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 12:55:10,041 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:55:10,041 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 12:55:10,042 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:55:10,042 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 12:55:10,046 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 12:55:10,047 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:55:10,047 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 12:55:10,048 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 12:57:15,556 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:57:15,557 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 12:57:15,566 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 12:57:15,574 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:57:15,574 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2012, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 12:57:15,576 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:57:15,576 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2012, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 12:57:46,649 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 12:57:46,649 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 12:57:46,649 - resource_logging.py:150 - __exit__ - DEBUG - Time: 31.06 seconds 2025-02-14 12:57:46,649 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:57:46,649 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26988.64 MB 2025-02-14 12:57:46,649 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34109.00 MB 2025-02-14 12:57:46,649 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7120.36 MB 2025-02-14 12:57:46,649 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53563.36 MB 2025-02-14 12:57:46,649 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40787.51 MB 2025-02-14 12:57:46,649 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12775.85 MB 2025-02-14 12:57:46,649 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43029.10 MB 2025-02-14 12:57:46,781 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 12:57:46,781 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 12:57:46,781 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 12:57:46,781 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:57:46,781 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34109.00 MB 2025-02-14 12:57:46,781 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26237.61 MB 2025-02-14 12:57:46,781 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7871.39 MB 2025-02-14 12:57:46,781 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40787.51 MB 2025-02-14 12:57:46,781 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 64583.89 MB 2025-02-14 12:57:46,781 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 23796.38 MB 2025-02-14 12:57:46,781 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54634.73 MB 2025-02-14 12:57:48,725 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 12:57:48,725 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 12:57:48,725 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-14 12:57:48,725 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:57:48,725 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26237.61 MB 2025-02-14 12:57:48,725 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26768.45 MB 2025-02-14 12:57:48,725 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 12:57:48,725 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64583.89 MB 2025-02-14 12:57:48,725 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30886.85 MB 2025-02-14 12:57:48,725 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33697.04 MB 2025-02-14 12:57:48,725 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30748.82 MB 2025-02-14 12:57:48,739 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 12:57:48,739 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 12:57:48,739 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:57:48,739 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:57:48,739 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26768.45 MB 2025-02-14 12:57:48,739 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28657.99 MB 2025-02-14 12:57:48,739 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 12:57:48,739 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30886.85 MB 2025-02-14 12:57:48,739 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32774.29 MB 2025-02-14 12:57:48,739 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 12:57:48,739 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30075.42 MB 2025-02-14 12:57:48,953 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 12:57:48,953 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 12:57:48,953 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 12:57:48,953 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:57:48,953 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28657.99 MB 2025-02-14 12:57:48,953 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30899.84 MB 2025-02-14 12:57:48,953 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 12:57:48,953 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32774.29 MB 2025-02-14 12:57:48,953 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38908.46 MB 2025-02-14 12:57:48,953 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 12:57:48,953 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36444.12 MB 2025-02-14 12:57:48,953 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 12:57:48,953 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 12:57:48,953 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 12:57:48,953 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:57:48,953 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26768.45 MB 2025-02-14 12:57:48,954 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30899.84 MB 2025-02-14 12:57:48,954 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 12:57:48,954 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30886.85 MB 2025-02-14 12:57:48,954 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38908.46 MB 2025-02-14 12:57:48,954 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-14 12:57:48,954 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36444.12 MB 2025-02-14 12:57:49,126 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 12:57:49,126 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 12:57:49,126 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 12:57:49,126 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:57:49,126 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32433.38 MB 2025-02-14 12:57:49,126 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33200.39 MB 2025-02-14 12:57:49,126 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 12:57:49,126 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38908.46 MB 2025-02-14 12:57:49,126 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39323.70 MB 2025-02-14 12:57:49,126 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 12:57:49,126 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33908.18 MB 2025-02-14 12:57:49,145 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 12:57:49,145 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 12:57:49,145 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:57:49,145 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:57:49,145 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33613.28 MB 2025-02-14 12:57:49,145 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33842.59 MB 2025-02-14 12:57:49,145 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.32 MB 2025-02-14 12:57:49,145 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39323.70 MB 2025-02-14 12:57:49,145 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39323.70 MB 2025-02-14 12:57:49,145 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:57:49,145 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34060.18 MB 2025-02-14 12:57:49,146 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 12:57:49,146 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 12:57:49,146 - resource_logging.py:150 - __exit__ - DEBUG - Time: 33.57 seconds 2025-02-14 12:57:49,146 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:57:49,146 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19978.68 MB 2025-02-14 12:57:49,146 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34043.30 MB 2025-02-14 12:57:49,146 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14064.62 MB 2025-02-14 12:57:49,146 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53563.36 MB 2025-02-14 12:57:49,146 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39323.70 MB 2025-02-14 12:57:49,146 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14239.66 MB 2025-02-14 12:57:49,146 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34060.18 MB 2025-02-14 12:57:49,417 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 12:57:49,417 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 12:57:49,417 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 12:57:49,417 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:57:49,417 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34043.30 MB 2025-02-14 12:57:49,417 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24977.35 MB 2025-02-14 12:57:49,417 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9065.95 MB 2025-02-14 12:57:49,417 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39323.70 MB 2025-02-14 12:57:49,417 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39323.70 MB 2025-02-14 12:57:49,417 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:57:49,417 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36550.36 MB 2025-02-14 12:57:49,435 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8147, cut from 8149 2025-02-14 12:57:49,435 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 12:57:49,441 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 12:57:49,441 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 12:57:49,441 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:57:49,441 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:57:49,441 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24977.35 MB 2025-02-14 12:57:49,441 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33400.56 MB 2025-02-14 12:57:49,441 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8423.21 MB 2025-02-14 12:57:49,441 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39323.70 MB 2025-02-14 12:57:49,441 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43511.71 MB 2025-02-14 12:57:49,441 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4188.01 MB 2025-02-14 12:57:49,441 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33400.56 MB 2025-02-14 12:57:49,604 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7939] 2025-02-14 12:57:49,605 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:57:49,605 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 12:57:49,606 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:57:49,606 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 12:57:49,611 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 12:57:49,612 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:57:49,612 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 12:57:49,612 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 12:58:40,952 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:58:40,953 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 12:58:40,961 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 12:58:40,969 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:58:40,969 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2492, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 12:58:40,971 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:58:40,971 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2492, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 12:59:19,778 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 12:59:19,778 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 12:59:19,778 - resource_logging.py:150 - __exit__ - DEBUG - Time: 38.79 seconds 2025-02-14 12:59:19,778 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:59:19,778 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30334.93 MB 2025-02-14 12:59:19,778 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39153.98 MB 2025-02-14 12:59:19,778 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8819.05 MB 2025-02-14 12:59:19,778 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 69256.35 MB 2025-02-14 12:59:19,778 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43100.67 MB 2025-02-14 12:59:19,778 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26155.68 MB 2025-02-14 12:59:19,778 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47973.03 MB 2025-02-14 12:59:19,945 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 12:59:19,945 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 12:59:19,945 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 12:59:19,945 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:59:19,945 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39153.98 MB 2025-02-14 12:59:19,945 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28733.77 MB 2025-02-14 12:59:19,945 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10420.21 MB 2025-02-14 12:59:19,945 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43100.67 MB 2025-02-14 12:59:19,945 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 75558.29 MB 2025-02-14 12:59:19,945 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 32457.62 MB 2025-02-14 12:59:19,945 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 64031.49 MB 2025-02-14 12:59:21,916 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 12:59:21,916 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 12:59:21,916 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.97 seconds 2025-02-14 12:59:21,916 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:59:21,916 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28733.77 MB 2025-02-14 12:59:21,916 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29264.61 MB 2025-02-14 12:59:21,916 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 12:59:21,916 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 75558.29 MB 2025-02-14 12:59:21,916 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31285.31 MB 2025-02-14 12:59:21,916 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -44272.98 MB 2025-02-14 12:59:21,916 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33244.98 MB 2025-02-14 12:59:21,930 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 12:59:21,930 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 12:59:21,930 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 12:59:21,930 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:59:21,930 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29264.61 MB 2025-02-14 12:59:21,930 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31154.14 MB 2025-02-14 12:59:21,930 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 12:59:21,930 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31285.31 MB 2025-02-14 12:59:21,930 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34588.33 MB 2025-02-14 12:59:21,930 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 12:59:21,930 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32571.57 MB 2025-02-14 12:59:22,141 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 12:59:22,141 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 12:59:22,141 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 12:59:22,141 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:59:22,142 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31154.14 MB 2025-02-14 12:59:22,142 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33396.00 MB 2025-02-14 12:59:22,142 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 12:59:22,142 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34588.33 MB 2025-02-14 12:59:22,142 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41194.36 MB 2025-02-14 12:59:22,142 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 12:59:22,142 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38940.28 MB 2025-02-14 12:59:22,142 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 12:59:22,142 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 12:59:22,142 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 12:59:22,142 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:59:22,142 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29264.61 MB 2025-02-14 12:59:22,142 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33396.00 MB 2025-02-14 12:59:22,142 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 12:59:22,142 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31285.31 MB 2025-02-14 12:59:22,142 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41194.36 MB 2025-02-14 12:59:22,142 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9909.04 MB 2025-02-14 12:59:22,142 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38940.28 MB 2025-02-14 12:59:22,314 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 12:59:22,314 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 12:59:22,314 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 12:59:22,314 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:59:22,314 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34929.54 MB 2025-02-14 12:59:22,314 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35696.54 MB 2025-02-14 12:59:22,314 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 12:59:22,314 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41194.36 MB 2025-02-14 12:59:22,314 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41607.50 MB 2025-02-14 12:59:22,314 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 12:59:22,314 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36404.33 MB 2025-02-14 12:59:22,334 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 12:59:22,334 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 12:59:22,334 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:59:22,334 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:59:22,334 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36109.43 MB 2025-02-14 12:59:22,334 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36338.49 MB 2025-02-14 12:59:22,334 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.06 MB 2025-02-14 12:59:22,334 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41607.50 MB 2025-02-14 12:59:22,334 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41607.50 MB 2025-02-14 12:59:22,334 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:59:22,334 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36551.84 MB 2025-02-14 12:59:22,335 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 12:59:22,335 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 12:59:22,335 - resource_logging.py:150 - __exit__ - DEBUG - Time: 41.36 seconds 2025-02-14 12:59:22,335 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:59:22,335 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21651.82 MB 2025-02-14 12:59:22,335 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36539.47 MB 2025-02-14 12:59:22,335 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14887.65 MB 2025-02-14 12:59:22,335 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60572.04 MB 2025-02-14 12:59:22,335 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41607.50 MB 2025-02-14 12:59:22,335 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18964.55 MB 2025-02-14 12:59:22,335 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36551.84 MB 2025-02-14 12:59:22,606 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 12:59:22,606 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 12:59:22,606 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 12:59:22,606 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:59:22,606 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36539.47 MB 2025-02-14 12:59:22,606 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26654.69 MB 2025-02-14 12:59:22,606 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9884.78 MB 2025-02-14 12:59:22,606 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41607.50 MB 2025-02-14 12:59:22,606 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41607.50 MB 2025-02-14 12:59:22,606 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 12:59:22,606 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39049.90 MB 2025-02-14 12:59:22,624 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8158, cut from 8160 2025-02-14 12:59:22,624 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 12:59:22,630 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 12:59:22,630 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 12:59:22,630 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 12:59:22,630 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 12:59:22,630 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26654.69 MB 2025-02-14 12:59:22,630 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35089.25 MB 2025-02-14 12:59:22,630 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8434.56 MB 2025-02-14 12:59:22,630 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41607.50 MB 2025-02-14 12:59:22,630 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45801.80 MB 2025-02-14 12:59:22,630 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4194.30 MB 2025-02-14 12:59:22,630 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35089.25 MB 2025-02-14 12:59:22,793 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7950] 2025-02-14 12:59:22,794 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:59:22,795 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 12:59:22,795 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:59:22,795 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 12:59:22,800 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 12:59:22,801 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 12:59:22,801 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 12:59:22,801 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 13:00:38,910 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:00:38,911 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 13:00:38,916 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 13:00:38,920 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:00:38,920 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1328, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 13:00:38,921 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:00:38,921 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1328, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 13:00:59,348 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 13:00:59,349 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 13:00:59,349 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.42 seconds 2025-02-14 13:00:59,349 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:00:59,349 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22222.42 MB 2025-02-14 13:00:59,349 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26922.14 MB 2025-02-14 13:00:59,349 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4699.72 MB 2025-02-14 13:00:59,349 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54186.21 MB 2025-02-14 13:00:59,349 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38463.86 MB 2025-02-14 13:00:59,349 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15722.35 MB 2025-02-14 13:00:59,349 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35770.66 MB 2025-02-14 13:00:59,420 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 13:00:59,420 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 13:00:59,420 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 13:00:59,420 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:00:59,421 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26922.14 MB 2025-02-14 13:00:59,421 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22681.71 MB 2025-02-14 13:00:59,421 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4240.43 MB 2025-02-14 13:00:59,421 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38463.86 MB 2025-02-14 13:00:59,421 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42251.32 MB 2025-02-14 13:00:59,421 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3787.46 MB 2025-02-14 13:00:59,421 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38719.76 MB 2025-02-14 13:01:01,346 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 13:01:01,346 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 13:01:01,346 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 13:01:01,346 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:01:01,346 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22681.71 MB 2025-02-14 13:01:01,346 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23212.55 MB 2025-02-14 13:01:01,346 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 13:01:01,346 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42251.32 MB 2025-02-14 13:01:01,346 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33764.15 MB 2025-02-14 13:01:01,346 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8487.17 MB 2025-02-14 13:01:01,346 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27191.88 MB 2025-02-14 13:01:01,360 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 13:01:01,360 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 13:01:01,360 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:01:01,360 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:01:01,360 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23212.55 MB 2025-02-14 13:01:01,360 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25102.08 MB 2025-02-14 13:01:01,360 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 13:01:01,360 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33764.15 MB 2025-02-14 13:01:01,360 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33764.15 MB 2025-02-14 13:01:01,360 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:01:01,360 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26519.51 MB 2025-02-14 13:01:01,574 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 13:01:01,574 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 13:01:01,574 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 13:01:01,574 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:01:01,574 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25102.08 MB 2025-02-14 13:01:01,574 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27343.94 MB 2025-02-14 13:01:01,574 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 13:01:01,574 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33764.15 MB 2025-02-14 13:01:01,574 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36123.44 MB 2025-02-14 13:01:01,574 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2359.30 MB 2025-02-14 13:01:01,574 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32888.22 MB 2025-02-14 13:01:01,575 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 13:01:01,575 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 13:01:01,575 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 13:01:01,575 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:01:01,575 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23212.55 MB 2025-02-14 13:01:01,575 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27343.94 MB 2025-02-14 13:01:01,575 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 13:01:01,575 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33764.15 MB 2025-02-14 13:01:01,575 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36123.44 MB 2025-02-14 13:01:01,575 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2359.30 MB 2025-02-14 13:01:01,575 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32888.22 MB 2025-02-14 13:01:01,745 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 13:01:01,745 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 13:01:01,745 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 13:01:01,745 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:01:01,745 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28877.48 MB 2025-02-14 13:01:01,745 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29644.48 MB 2025-02-14 13:01:01,745 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 13:01:01,745 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36123.44 MB 2025-02-14 13:01:01,745 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36536.58 MB 2025-02-14 13:01:01,745 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 13:01:01,745 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30352.27 MB 2025-02-14 13:01:01,764 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 13:01:01,764 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 13:01:01,764 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:01:01,764 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:01:01,764 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30057.37 MB 2025-02-14 13:01:01,764 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30284.20 MB 2025-02-14 13:01:01,764 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.82 MB 2025-02-14 13:01:01,764 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36536.58 MB 2025-02-14 13:01:01,764 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36536.58 MB 2025-02-14 13:01:01,764 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:01:01,764 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30517.21 MB 2025-02-14 13:01:01,766 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 13:01:01,766 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 13:01:01,766 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.84 seconds 2025-02-14 13:01:01,766 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:01:01,766 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17595.56 MB 2025-02-14 13:01:01,766 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30484.60 MB 2025-02-14 13:01:01,766 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12889.04 MB 2025-02-14 13:01:01,766 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54186.21 MB 2025-02-14 13:01:01,766 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36536.58 MB 2025-02-14 13:01:01,766 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17649.63 MB 2025-02-14 13:01:01,766 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30517.21 MB 2025-02-14 13:01:02,034 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 13:01:02,034 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 13:01:02,034 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 13:01:02,034 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:01:02,034 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30484.60 MB 2025-02-14 13:01:02,034 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22589.67 MB 2025-02-14 13:01:02,034 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7894.94 MB 2025-02-14 13:01:02,034 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36536.58 MB 2025-02-14 13:01:02,034 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36536.58 MB 2025-02-14 13:01:02,035 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:01:02,035 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32987.98 MB 2025-02-14 13:01:02,052 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8135, cut from 8137 2025-02-14 13:01:02,052 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 13:01:02,059 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 13:01:02,059 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 13:01:02,059 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:01:02,059 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:01:02,059 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22589.67 MB 2025-02-14 13:01:02,059 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31000.49 MB 2025-02-14 13:01:02,059 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8410.82 MB 2025-02-14 13:01:02,059 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36536.58 MB 2025-02-14 13:01:02,059 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40718.30 MB 2025-02-14 13:01:02,059 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4181.72 MB 2025-02-14 13:01:02,059 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31000.49 MB 2025-02-14 13:01:02,220 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7927] 2025-02-14 13:01:02,222 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:01:02,222 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 13:01:02,223 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:01:02,223 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 13:01:02,228 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 13:01:02,229 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:01:02,229 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 13:01:02,229 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 13:01:58,269 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:01:58,269 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 13:01:58,274 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 13:01:58,278 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:01:58,278 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1631, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 13:01:58,279 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:01:58,279 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1631, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 13:02:23,515 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 13:02:23,515 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 13:02:23,515 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.23 seconds 2025-02-14 13:02:23,515 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:02:23,515 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24333.78 MB 2025-02-14 13:02:23,515 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30105.79 MB 2025-02-14 13:02:23,515 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5772.02 MB 2025-02-14 13:02:23,515 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49081.75 MB 2025-02-14 13:02:23,515 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39420.17 MB 2025-02-14 13:02:23,515 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9661.58 MB 2025-02-14 13:02:23,515 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39015.28 MB 2025-02-14 13:02:23,611 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 13:02:23,612 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 13:02:23,612 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 13:02:23,612 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:02:23,612 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30105.79 MB 2025-02-14 13:02:23,612 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24256.91 MB 2025-02-14 13:02:23,612 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5848.88 MB 2025-02-14 13:02:23,612 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39420.17 MB 2025-02-14 13:02:23,612 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55320.77 MB 2025-02-14 13:02:23,612 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 15900.61 MB 2025-02-14 13:02:23,612 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46784.08 MB 2025-02-14 13:02:25,543 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 13:02:25,543 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 13:02:25,543 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 13:02:25,543 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:02:25,543 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24256.91 MB 2025-02-14 13:02:25,544 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24787.75 MB 2025-02-14 13:02:25,544 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 13:02:25,544 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55320.77 MB 2025-02-14 13:02:25,544 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30880.56 MB 2025-02-14 13:02:25,544 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24440.21 MB 2025-02-14 13:02:25,544 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28767.09 MB 2025-02-14 13:02:25,560 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 13:02:25,560 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 13:02:25,560 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:02:25,560 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:02:25,560 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24787.75 MB 2025-02-14 13:02:25,560 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26677.29 MB 2025-02-14 13:02:25,560 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 13:02:25,560 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30880.56 MB 2025-02-14 13:02:25,560 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31824.28 MB 2025-02-14 13:02:25,560 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 13:02:25,560 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28094.72 MB 2025-02-14 13:02:25,770 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 13:02:25,770 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 13:02:25,770 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 13:02:25,770 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:02:25,770 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26677.29 MB 2025-02-14 13:02:25,770 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28919.14 MB 2025-02-14 13:02:25,770 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 13:02:25,770 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31824.28 MB 2025-02-14 13:02:25,770 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37486.59 MB 2025-02-14 13:02:25,770 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 13:02:25,770 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34463.42 MB 2025-02-14 13:02:25,770 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 13:02:25,770 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 13:02:25,770 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 13:02:25,770 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:02:25,770 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24787.75 MB 2025-02-14 13:02:25,770 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28919.14 MB 2025-02-14 13:02:25,771 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 13:02:25,771 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30880.56 MB 2025-02-14 13:02:25,771 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37486.59 MB 2025-02-14 13:02:25,771 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 13:02:25,771 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34463.42 MB 2025-02-14 13:02:25,938 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 13:02:25,938 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 13:02:25,938 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 13:02:25,938 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:02:25,938 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30452.69 MB 2025-02-14 13:02:25,938 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31219.69 MB 2025-02-14 13:02:25,938 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 13:02:25,938 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37486.59 MB 2025-02-14 13:02:25,938 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37901.83 MB 2025-02-14 13:02:25,938 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 13:02:25,938 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31927.48 MB 2025-02-14 13:02:25,957 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 13:02:25,957 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 13:02:25,957 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:02:25,957 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:02:25,957 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31632.58 MB 2025-02-14 13:02:25,957 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31862.42 MB 2025-02-14 13:02:25,957 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.85 MB 2025-02-14 13:02:25,957 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37901.83 MB 2025-02-14 13:02:25,957 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37901.83 MB 2025-02-14 13:02:25,957 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:02:25,957 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32073.53 MB 2025-02-14 13:02:25,958 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 13:02:25,958 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 13:02:25,958 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.68 seconds 2025-02-14 13:02:25,958 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:02:25,958 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18651.24 MB 2025-02-14 13:02:25,958 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32063.42 MB 2025-02-14 13:02:25,958 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13412.18 MB 2025-02-14 13:02:25,958 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49081.75 MB 2025-02-14 13:02:25,958 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37901.83 MB 2025-02-14 13:02:25,958 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11179.92 MB 2025-02-14 13:02:25,958 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32073.53 MB 2025-02-14 13:02:26,227 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 13:02:26,228 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 13:02:26,228 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 13:02:26,228 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:02:26,228 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32063.42 MB 2025-02-14 13:02:26,228 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23654.49 MB 2025-02-14 13:02:26,228 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8408.93 MB 2025-02-14 13:02:26,228 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37901.83 MB 2025-02-14 13:02:26,228 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37901.83 MB 2025-02-14 13:02:26,228 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:02:26,228 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34574.17 MB 2025-02-14 13:02:26,246 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8159, cut from 8161 2025-02-14 13:02:26,246 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 13:02:26,252 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 13:02:26,252 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 13:02:26,252 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:02:26,252 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:02:26,252 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23654.49 MB 2025-02-14 13:02:26,252 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32090.08 MB 2025-02-14 13:02:26,252 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8435.59 MB 2025-02-14 13:02:26,252 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37901.83 MB 2025-02-14 13:02:26,252 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46290.44 MB 2025-02-14 13:02:26,252 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8388.61 MB 2025-02-14 13:02:26,252 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32090.08 MB 2025-02-14 13:02:26,404 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7951] 2025-02-14 13:02:26,406 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:02:26,406 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 13:02:26,407 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:02:26,407 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 13:02:26,411 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 13:02:26,412 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:02:26,412 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 13:02:26,412 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 13:04:01,767 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:04:01,768 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 13:04:01,773 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 13:04:01,778 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:04:01,778 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1250, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 13:04:01,779 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:04:01,780 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1250, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 13:04:20,939 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 13:04:20,940 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 13:04:20,940 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.15 seconds 2025-02-14 13:04:20,940 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:04:20,940 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21678.91 MB 2025-02-14 13:04:20,940 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26102.59 MB 2025-02-14 13:04:20,940 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4423.68 MB 2025-02-14 13:04:20,940 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54679.04 MB 2025-02-14 13:04:20,940 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38086.38 MB 2025-02-14 13:04:20,940 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16592.67 MB 2025-02-14 13:04:20,940 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35000.65 MB 2025-02-14 13:04:21,013 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 13:04:21,013 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 13:04:21,013 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 13:04:21,013 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:04:21,013 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26102.59 MB 2025-02-14 13:04:21,013 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22276.21 MB 2025-02-14 13:04:21,013 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3826.38 MB 2025-02-14 13:04:21,013 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38086.38 MB 2025-02-14 13:04:21,013 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46810.53 MB 2025-02-14 13:04:21,013 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8724.15 MB 2025-02-14 13:04:21,013 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39228.06 MB 2025-02-14 13:04:22,923 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 13:04:22,923 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 13:04:22,923 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 13:04:22,923 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:04:22,923 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22276.21 MB 2025-02-14 13:04:22,923 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22807.05 MB 2025-02-14 13:04:22,923 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 13:04:22,923 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46810.53 MB 2025-02-14 13:04:22,923 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29479.67 MB 2025-02-14 13:04:22,923 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17330.86 MB 2025-02-14 13:04:22,923 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26786.39 MB 2025-02-14 13:04:22,937 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 13:04:22,937 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 13:04:22,937 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:04:22,937 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:04:22,937 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22807.05 MB 2025-02-14 13:04:22,937 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24696.59 MB 2025-02-14 13:04:22,937 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 13:04:22,937 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29479.67 MB 2025-02-14 13:04:22,937 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29479.67 MB 2025-02-14 13:04:22,937 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:04:22,937 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26114.02 MB 2025-02-14 13:04:23,145 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 13:04:23,146 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 13:04:23,146 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 13:04:23,146 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:04:23,146 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24696.59 MB 2025-02-14 13:04:23,146 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26938.44 MB 2025-02-14 13:04:23,146 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 13:04:23,146 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29479.67 MB 2025-02-14 13:04:23,146 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35141.98 MB 2025-02-14 13:04:23,146 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 13:04:23,146 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32482.72 MB 2025-02-14 13:04:23,146 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 13:04:23,146 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 13:04:23,146 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 13:04:23,146 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:04:23,146 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22807.05 MB 2025-02-14 13:04:23,146 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26938.44 MB 2025-02-14 13:04:23,146 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 13:04:23,146 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29479.67 MB 2025-02-14 13:04:23,146 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35141.98 MB 2025-02-14 13:04:23,146 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 13:04:23,146 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32482.72 MB 2025-02-14 13:04:23,310 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 13:04:23,310 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 13:04:23,310 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 13:04:23,310 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:04:23,310 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28471.98 MB 2025-02-14 13:04:23,310 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29238.99 MB 2025-02-14 13:04:23,310 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 13:04:23,310 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35141.98 MB 2025-02-14 13:04:23,310 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35555.12 MB 2025-02-14 13:04:23,310 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 13:04:23,310 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29946.78 MB 2025-02-14 13:04:23,329 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 13:04:23,329 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 13:04:23,329 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:04:23,329 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:04:23,329 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29651.88 MB 2025-02-14 13:04:23,329 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29880.76 MB 2025-02-14 13:04:23,329 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.89 MB 2025-02-14 13:04:23,329 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35555.12 MB 2025-02-14 13:04:23,329 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35555.12 MB 2025-02-14 13:04:23,329 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:04:23,329 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30117.22 MB 2025-02-14 13:04:23,330 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 13:04:23,330 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 13:04:23,330 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.55 seconds 2025-02-14 13:04:23,330 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:04:23,330 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17323.81 MB 2025-02-14 13:04:23,330 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30081.56 MB 2025-02-14 13:04:23,330 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12757.76 MB 2025-02-14 13:04:23,330 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54679.04 MB 2025-02-14 13:04:23,331 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35555.12 MB 2025-02-14 13:04:23,331 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19123.93 MB 2025-02-14 13:04:23,331 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30117.22 MB 2025-02-14 13:04:23,600 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 13:04:23,600 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 13:04:23,600 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 13:04:23,600 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:04:23,600 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30081.56 MB 2025-02-14 13:04:23,600 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22324.01 MB 2025-02-14 13:04:23,600 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7757.56 MB 2025-02-14 13:04:23,600 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35555.12 MB 2025-02-14 13:04:23,600 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35555.12 MB 2025-02-14 13:04:23,600 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:04:23,600 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32589.85 MB 2025-02-14 13:04:23,618 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8151, cut from 8153 2025-02-14 13:04:23,618 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-14 13:04:23,625 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 13:04:23,625 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 13:04:23,625 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:04:23,625 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:04:23,625 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22324.01 MB 2025-02-14 13:04:23,625 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30751.88 MB 2025-02-14 13:04:23,625 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8427.88 MB 2025-02-14 13:04:23,625 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35555.12 MB 2025-02-14 13:04:23,625 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43935.33 MB 2025-02-14 13:04:23,625 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8380.22 MB 2025-02-14 13:04:23,625 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30751.88 MB 2025-02-14 13:04:23,782 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7943] 2025-02-14 13:04:23,783 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:04:23,783 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 13:04:23,784 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:04:23,784 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 13:04:23,789 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 13:04:23,790 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:04:23,790 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 13:04:23,790 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-14 13:04:33,807 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:04:33,807 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 13:04:33,811 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 13:04:33,815 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:04:33,815 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2225, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 13:04:33,816 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:04:33,816 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2225, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 13:05:08,545 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 13:05:08,545 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 13:05:08,545 - resource_logging.py:150 - __exit__ - DEBUG - Time: 34.72 seconds 2025-02-14 13:05:08,545 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:05:08,545 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28472.86 MB 2025-02-14 13:05:08,545 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36347.67 MB 2025-02-14 13:05:08,545 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7874.81 MB 2025-02-14 13:05:08,546 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52315.55 MB 2025-02-14 13:05:08,546 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41536.19 MB 2025-02-14 13:05:08,546 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10779.36 MB 2025-02-14 13:05:08,546 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45192.80 MB 2025-02-14 13:05:08,691 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 13:05:08,691 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 13:05:08,691 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-14 13:05:08,691 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:05:08,691 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36347.67 MB 2025-02-14 13:05:08,691 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27344.93 MB 2025-02-14 13:05:08,691 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9002.74 MB 2025-02-14 13:05:08,691 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41536.19 MB 2025-02-14 13:05:08,691 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 69642.22 MB 2025-02-14 13:05:08,691 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 28106.03 MB 2025-02-14 13:05:08,691 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 58904.38 MB 2025-02-14 13:05:10,654 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 13:05:10,654 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 13:05:10,654 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.96 seconds 2025-02-14 13:05:10,654 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:05:10,655 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27344.93 MB 2025-02-14 13:05:10,655 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27875.77 MB 2025-02-14 13:05:10,655 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 13:05:10,655 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 69642.22 MB 2025-02-14 13:05:10,655 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30886.85 MB 2025-02-14 13:05:10,655 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -38755.37 MB 2025-02-14 13:05:10,655 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31855.11 MB 2025-02-14 13:05:10,669 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 13:05:10,669 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 13:05:10,669 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:05:10,669 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:05:10,669 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27875.77 MB 2025-02-14 13:05:10,669 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29765.31 MB 2025-02-14 13:05:10,669 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 13:05:10,669 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30886.85 MB 2025-02-14 13:05:10,669 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34189.87 MB 2025-02-14 13:05:10,669 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 13:05:10,669 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31182.74 MB 2025-02-14 13:05:10,882 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 13:05:10,883 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 13:05:10,883 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 13:05:10,883 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:05:10,883 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29765.31 MB 2025-02-14 13:05:10,883 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32007.16 MB 2025-02-14 13:05:10,883 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 13:05:10,883 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34189.87 MB 2025-02-14 13:05:10,883 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40324.04 MB 2025-02-14 13:05:10,883 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 13:05:10,883 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37551.44 MB 2025-02-14 13:05:10,883 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 13:05:10,883 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 13:05:10,883 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 13:05:10,883 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:05:10,883 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27875.77 MB 2025-02-14 13:05:10,883 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32007.16 MB 2025-02-14 13:05:10,883 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 13:05:10,883 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30886.85 MB 2025-02-14 13:05:10,883 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40324.04 MB 2025-02-14 13:05:10,883 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9437.18 MB 2025-02-14 13:05:10,883 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37551.44 MB 2025-02-14 13:05:11,053 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 13:05:11,053 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 13:05:11,053 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 13:05:11,053 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:05:11,053 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33540.71 MB 2025-02-14 13:05:11,053 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34307.71 MB 2025-02-14 13:05:11,053 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 13:05:11,053 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40324.04 MB 2025-02-14 13:05:11,053 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40739.27 MB 2025-02-14 13:05:11,053 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 13:05:11,053 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35015.50 MB 2025-02-14 13:05:11,072 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 13:05:11,072 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 13:05:11,072 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:05:11,072 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:05:11,072 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34720.60 MB 2025-02-14 13:05:11,072 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34949.43 MB 2025-02-14 13:05:11,072 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.84 MB 2025-02-14 13:05:11,072 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40739.27 MB 2025-02-14 13:05:11,072 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40739.27 MB 2025-02-14 13:05:11,072 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:05:11,072 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35171.03 MB 2025-02-14 13:05:11,074 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 13:05:11,074 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 13:05:11,074 - resource_logging.py:150 - __exit__ - DEBUG - Time: 37.26 seconds 2025-02-14 13:05:11,074 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:05:11,074 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20720.78 MB 2025-02-14 13:05:11,074 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35150.19 MB 2025-02-14 13:05:11,074 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14429.40 MB 2025-02-14 13:05:11,074 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52315.55 MB 2025-02-14 13:05:11,074 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40739.27 MB 2025-02-14 13:05:11,074 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11576.28 MB 2025-02-14 13:05:11,074 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35171.03 MB 2025-02-14 13:05:11,345 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 13:05:11,345 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 13:05:11,345 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 13:05:11,345 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:05:11,345 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35150.19 MB 2025-02-14 13:05:11,345 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25720.22 MB 2025-02-14 13:05:11,345 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9429.97 MB 2025-02-14 13:05:11,345 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40739.27 MB 2025-02-14 13:05:11,345 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40739.27 MB 2025-02-14 13:05:11,345 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:05:11,345 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37657.86 MB 2025-02-14 13:05:11,363 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8149, cut from 8151 2025-02-14 13:05:11,363 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 13:05:11,369 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 13:05:11,369 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 13:05:11,369 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:05:11,369 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:05:11,369 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25720.22 MB 2025-02-14 13:05:11,369 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34146.40 MB 2025-02-14 13:05:11,369 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8426.18 MB 2025-02-14 13:05:11,369 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40739.27 MB 2025-02-14 13:05:11,369 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49115.30 MB 2025-02-14 13:05:11,369 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8376.03 MB 2025-02-14 13:05:11,369 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34146.40 MB 2025-02-14 13:05:11,532 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7941] 2025-02-14 13:05:11,534 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:05:11,534 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 13:05:11,535 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:05:11,535 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 13:05:11,539 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 13:05:11,540 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:05:11,540 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 13:05:11,541 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 13:05:22,525 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:05:22,525 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 13:05:22,530 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 13:05:22,534 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:05:22,534 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 174, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 13:05:22,535 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:05:22,535 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 174, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 13:05:25,278 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 13:05:25,278 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 13:05:25,278 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.74 seconds 2025-02-14 13:05:25,278 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:05:25,278 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14181.17 MB 2025-02-14 13:05:25,278 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14796.94 MB 2025-02-14 13:05:25,278 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 615.78 MB 2025-02-14 13:05:25,278 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57491.32 MB 2025-02-14 13:05:25,278 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 16911.43 MB 2025-02-14 13:05:25,278 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -40579.89 MB 2025-02-14 13:05:25,278 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23653.34 MB 2025-02-14 13:05:25,292 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 13:05:25,292 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 13:05:25,292 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:05:25,292 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:05:25,292 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14796.94 MB 2025-02-14 13:05:25,292 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15005.41 MB 2025-02-14 13:05:25,292 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 208.47 MB 2025-02-14 13:05:25,292 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 16911.43 MB 2025-02-14 13:05:25,292 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18329.11 MB 2025-02-14 13:05:25,292 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1417.67 MB 2025-02-14 13:05:25,292 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17110.05 MB 2025-02-14 13:05:26,077 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 13:05:26,078 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 13:05:26,078 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.78 seconds 2025-02-14 13:05:26,078 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:05:26,078 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15005.41 MB 2025-02-14 13:05:26,078 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15219.08 MB 2025-02-14 13:05:26,078 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 213.66 MB 2025-02-14 13:05:26,078 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18329.11 MB 2025-02-14 13:05:26,078 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17758.68 MB 2025-02-14 13:05:26,078 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -570.43 MB 2025-02-14 13:05:26,078 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19176.89 MB 2025-02-14 13:05:26,086 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 13:05:26,086 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 13:05:26,086 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:05:26,086 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:05:26,086 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15219.01 MB 2025-02-14 13:05:26,086 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15979.36 MB 2025-02-14 13:05:26,086 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 760.35 MB 2025-02-14 13:05:26,086 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17758.68 MB 2025-02-14 13:05:26,086 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18140.36 MB 2025-02-14 13:05:26,086 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 381.68 MB 2025-02-14 13:05:26,086 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16549.88 MB 2025-02-14 13:05:26,176 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 13:05:26,176 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 13:05:26,176 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 13:05:26,176 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:05:26,176 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15979.36 MB 2025-02-14 13:05:26,176 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16881.75 MB 2025-02-14 13:05:26,176 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 902.39 MB 2025-02-14 13:05:26,176 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18140.36 MB 2025-02-14 13:05:26,176 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20239.61 MB 2025-02-14 13:05:26,176 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2099.25 MB 2025-02-14 13:05:26,176 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19114.20 MB 2025-02-14 13:05:26,177 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 13:05:26,177 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 13:05:26,177 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 13:05:26,177 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:05:26,177 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15219.01 MB 2025-02-14 13:05:26,177 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16881.75 MB 2025-02-14 13:05:26,177 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1662.74 MB 2025-02-14 13:05:26,177 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17758.68 MB 2025-02-14 13:05:26,177 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20239.61 MB 2025-02-14 13:05:26,177 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2480.93 MB 2025-02-14 13:05:26,177 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19114.20 MB 2025-02-14 13:05:26,246 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 13:05:26,246 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 13:05:26,246 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 13:05:26,246 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:05:26,246 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17499.00 MB 2025-02-14 13:05:26,246 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17808.64 MB 2025-02-14 13:05:26,246 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 309.64 MB 2025-02-14 13:05:26,246 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20239.61 MB 2025-02-14 13:05:26,246 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20401.09 MB 2025-02-14 13:05:26,247 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 161.48 MB 2025-02-14 13:05:26,247 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18103.35 MB 2025-02-14 13:05:26,256 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 13:05:26,256 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 13:05:26,256 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:05:26,256 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:05:26,256 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17974.83 MB 2025-02-14 13:05:26,256 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18202.04 MB 2025-02-14 13:05:26,256 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.21 MB 2025-02-14 13:05:26,256 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20401.09 MB 2025-02-14 13:05:26,256 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20401.09 MB 2025-02-14 13:05:26,256 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:05:26,256 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18224.55 MB 2025-02-14 13:05:26,257 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 13:05:26,257 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 13:05:26,257 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.72 seconds 2025-02-14 13:05:26,257 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:05:26,257 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13574.94 MB 2025-02-14 13:05:26,257 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18403.11 MB 2025-02-14 13:05:26,257 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4828.17 MB 2025-02-14 13:05:26,257 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57491.32 MB 2025-02-14 13:05:26,257 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20401.09 MB 2025-02-14 13:05:26,257 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -37090.23 MB 2025-02-14 13:05:26,257 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18403.11 MB 2025-02-14 13:05:26,526 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 13:05:26,526 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 13:05:26,526 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 13:05:26,526 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:05:26,526 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18403.11 MB 2025-02-14 13:05:26,526 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17452.07 MB 2025-02-14 13:05:26,526 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -951.04 MB 2025-02-14 13:05:26,526 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20401.09 MB 2025-02-14 13:05:26,526 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20401.09 MB 2025-02-14 13:05:26,527 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:05:26,527 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19206.84 MB 2025-02-14 13:05:26,544 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 13:05:26,545 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2,'] 2025-02-14 13:05:26,551 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 13:05:26,551 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 13:05:26,551 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:05:26,551 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:05:26,551 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17452.07 MB 2025-02-14 13:05:26,551 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25891.09 MB 2025-02-14 13:05:26,551 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 13:05:26,551 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20401.09 MB 2025-02-14 13:05:26,551 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30891.05 MB 2025-02-14 13:05:26,551 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 13:05:26,551 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25891.09 MB 2025-02-14 13:05:26,713 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 13:05:26,714 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:05:26,714 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 13:05:26,715 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:05:26,715 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 13:05:26,720 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 13:05:26,721 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:05:26,721 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 13:05:26,721 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2,'] 2025-02-14 13:06:50,377 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:06:50,378 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 13:06:50,383 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 13:06:50,388 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:06:50,388 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 197, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 13:06:50,389 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:06:50,389 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 197, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 13:06:53,450 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 13:06:53,450 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 13:06:53,450 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.06 seconds 2025-02-14 13:06:53,450 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:06:53,450 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14341.43 MB 2025-02-14 13:06:53,450 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15038.61 MB 2025-02-14 13:06:53,450 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 697.17 MB 2025-02-14 13:06:53,450 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43476.06 MB 2025-02-14 13:06:53,450 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18438.16 MB 2025-02-14 13:06:53,450 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25037.90 MB 2025-02-14 13:06:53,450 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24040.10 MB 2025-02-14 13:06:53,466 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 13:06:53,466 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 13:06:53,466 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:06:53,466 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:06:53,466 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15038.61 MB 2025-02-14 13:06:53,466 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15257.52 MB 2025-02-14 13:06:53,466 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 218.91 MB 2025-02-14 13:06:53,466 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18438.16 MB 2025-02-14 13:06:53,466 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19075.69 MB 2025-02-14 13:06:53,466 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 637.53 MB 2025-02-14 13:06:53,466 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17585.18 MB 2025-02-14 13:06:54,325 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 13:06:54,325 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 13:06:54,325 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.86 seconds 2025-02-14 13:06:54,325 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:06:54,325 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15257.52 MB 2025-02-14 13:06:54,325 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15496.40 MB 2025-02-14 13:06:54,325 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 238.88 MB 2025-02-14 13:06:54,325 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19075.69 MB 2025-02-14 13:06:54,325 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19096.67 MB 2025-02-14 13:06:54,325 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 20.97 MB 2025-02-14 13:06:54,325 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19428.99 MB 2025-02-14 13:06:54,333 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 13:06:54,333 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 13:06:54,333 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:06:54,333 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:06:54,333 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15496.33 MB 2025-02-14 13:06:54,333 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16346.41 MB 2025-02-14 13:06:54,333 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 850.08 MB 2025-02-14 13:06:54,333 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19096.67 MB 2025-02-14 13:06:54,333 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19096.67 MB 2025-02-14 13:06:54,333 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:06:54,333 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16984.26 MB 2025-02-14 13:06:54,433 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 13:06:54,433 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 13:06:54,433 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 13:06:54,433 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:06:54,433 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16346.41 MB 2025-02-14 13:06:54,433 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17355.28 MB 2025-02-14 13:06:54,433 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1008.87 MB 2025-02-14 13:06:54,433 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19096.67 MB 2025-02-14 13:06:54,433 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21013.46 MB 2025-02-14 13:06:54,433 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1916.80 MB 2025-02-14 13:06:54,433 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19854.37 MB 2025-02-14 13:06:54,434 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 13:06:54,434 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 13:06:54,434 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 13:06:54,434 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:06:54,434 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15496.33 MB 2025-02-14 13:06:54,434 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17355.28 MB 2025-02-14 13:06:54,434 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1858.95 MB 2025-02-14 13:06:54,434 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19096.67 MB 2025-02-14 13:06:54,434 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21013.46 MB 2025-02-14 13:06:54,434 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1916.80 MB 2025-02-14 13:06:54,434 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19854.37 MB 2025-02-14 13:06:54,513 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 13:06:54,513 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 13:06:54,513 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 13:06:54,513 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:06:54,513 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18045.38 MB 2025-02-14 13:06:54,513 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18391.58 MB 2025-02-14 13:06:54,513 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 346.20 MB 2025-02-14 13:06:54,513 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21013.46 MB 2025-02-14 13:06:54,513 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21195.92 MB 2025-02-14 13:06:54,513 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 182.45 MB 2025-02-14 13:06:54,513 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18715.67 MB 2025-02-14 13:06:54,524 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 13:06:54,524 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 13:06:54,524 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:06:54,524 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:06:54,524 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18577.39 MB 2025-02-14 13:06:54,524 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18790.13 MB 2025-02-14 13:06:54,524 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 212.75 MB 2025-02-14 13:06:54,524 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21195.92 MB 2025-02-14 13:06:54,524 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21195.92 MB 2025-02-14 13:06:54,524 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:06:54,524 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18826.59 MB 2025-02-14 13:06:54,525 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 13:06:54,525 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 13:06:54,525 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.13 seconds 2025-02-14 13:06:54,525 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:06:54,525 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13655.07 MB 2025-02-14 13:06:54,525 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18990.96 MB 2025-02-14 13:06:54,525 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5335.89 MB 2025-02-14 13:06:54,525 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43476.06 MB 2025-02-14 13:06:54,525 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21195.92 MB 2025-02-14 13:06:54,525 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22280.14 MB 2025-02-14 13:06:54,525 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18990.96 MB 2025-02-14 13:06:54,793 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 13:06:54,793 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 13:06:54,793 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 13:06:54,793 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:06:54,793 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18990.96 MB 2025-02-14 13:06:54,793 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17618.42 MB 2025-02-14 13:06:54,793 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1372.54 MB 2025-02-14 13:06:54,793 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21195.92 MB 2025-02-14 13:06:54,793 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21195.92 MB 2025-02-14 13:06:54,793 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:06:54,793 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18990.97 MB 2025-02-14 13:06:54,811 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8152, cut from 8154 2025-02-14 13:06:54,812 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2.'] 2025-02-14 13:06:54,818 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 13:06:54,818 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 13:06:54,818 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:06:54,818 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:06:54,818 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17618.42 MB 2025-02-14 13:06:54,818 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26047.55 MB 2025-02-14 13:06:54,818 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8429.12 MB 2025-02-14 13:06:54,818 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21195.92 MB 2025-02-14 13:06:54,818 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31671.19 MB 2025-02-14 13:06:54,818 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10475.27 MB 2025-02-14 13:06:54,818 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26047.55 MB 2025-02-14 13:06:54,983 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7944] 2025-02-14 13:06:54,984 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:06:54,984 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 13:06:54,985 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:06:54,985 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 13:06:54,990 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 13:06:54,991 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:06:54,991 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 13:06:54,991 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2.'] 2025-02-14 13:07:57,948 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:07:57,948 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 13:07:57,952 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 13:07:57,956 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:07:57,956 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1790, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 13:07:57,957 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:07:57,957 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1790, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 13:08:25,448 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 13:08:25,448 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 13:08:25,448 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.48 seconds 2025-02-14 13:08:25,448 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:08:25,448 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25441.71 MB 2025-02-14 13:08:25,448 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31777.21 MB 2025-02-14 13:08:25,448 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6335.50 MB 2025-02-14 13:08:25,448 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40051.41 MB 2025-02-14 13:08:25,448 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40005.27 MB 2025-02-14 13:08:25,448 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -46.14 MB 2025-02-14 13:08:25,448 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40576.20 MB 2025-02-14 13:08:25,558 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 13:08:25,558 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 13:08:25,558 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 13:08:25,558 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:08:25,558 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31777.21 MB 2025-02-14 13:08:25,558 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25083.50 MB 2025-02-14 13:08:25,558 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6693.71 MB 2025-02-14 13:08:25,558 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40005.27 MB 2025-02-14 13:08:25,558 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 59250.84 MB 2025-02-14 13:08:25,558 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 19245.56 MB 2025-02-14 13:08:25,558 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50091.17 MB 2025-02-14 13:08:27,484 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 13:08:27,484 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 13:08:27,484 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 13:08:27,484 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:08:27,484 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25083.50 MB 2025-02-14 13:08:27,484 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25614.34 MB 2025-02-14 13:08:27,484 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 13:08:27,484 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59250.84 MB 2025-02-14 13:08:27,484 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35085.35 MB 2025-02-14 13:08:27,484 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24165.48 MB 2025-02-14 13:08:27,484 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29593.68 MB 2025-02-14 13:08:27,498 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 13:08:27,498 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 13:08:27,498 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:08:27,498 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:08:27,498 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25614.34 MB 2025-02-14 13:08:27,498 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27503.88 MB 2025-02-14 13:08:27,498 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 13:08:27,498 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35085.35 MB 2025-02-14 13:08:27,498 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35085.35 MB 2025-02-14 13:08:27,498 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:08:27,498 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28921.31 MB 2025-02-14 13:08:27,713 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 13:08:27,713 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 13:08:27,713 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 13:08:27,713 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:08:27,713 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27503.88 MB 2025-02-14 13:08:27,713 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29745.73 MB 2025-02-14 13:08:27,713 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 13:08:27,713 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35085.35 MB 2025-02-14 13:08:27,713 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38388.37 MB 2025-02-14 13:08:27,713 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 13:08:27,713 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35290.02 MB 2025-02-14 13:08:27,713 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 13:08:27,713 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 13:08:27,713 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 13:08:27,713 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:08:27,713 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25614.34 MB 2025-02-14 13:08:27,713 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29745.73 MB 2025-02-14 13:08:27,713 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 13:08:27,713 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35085.35 MB 2025-02-14 13:08:27,714 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38388.37 MB 2025-02-14 13:08:27,714 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 13:08:27,714 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35290.02 MB 2025-02-14 13:08:27,886 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 13:08:27,886 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 13:08:27,886 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 13:08:27,886 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:08:27,886 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31279.28 MB 2025-02-14 13:08:27,886 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32046.28 MB 2025-02-14 13:08:27,886 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 13:08:27,886 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38388.37 MB 2025-02-14 13:08:27,886 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38803.60 MB 2025-02-14 13:08:27,886 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 13:08:27,886 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32754.07 MB 2025-02-14 13:08:27,907 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 13:08:27,907 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 13:08:27,907 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:08:27,907 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:08:27,907 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32459.17 MB 2025-02-14 13:08:27,907 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32689.15 MB 2025-02-14 13:08:27,907 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.98 MB 2025-02-14 13:08:27,907 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38803.60 MB 2025-02-14 13:08:27,907 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38803.60 MB 2025-02-14 13:08:27,907 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:08:27,907 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32903.08 MB 2025-02-14 13:08:27,908 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 13:08:27,908 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 13:08:27,908 - resource_logging.py:150 - __exit__ - DEBUG - Time: 29.95 seconds 2025-02-14 13:08:27,908 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:08:27,909 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19205.21 MB 2025-02-14 13:08:27,909 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32890.07 MB 2025-02-14 13:08:27,909 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13684.86 MB 2025-02-14 13:08:27,909 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40051.41 MB 2025-02-14 13:08:27,909 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38803.60 MB 2025-02-14 13:08:27,909 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1247.81 MB 2025-02-14 13:08:27,909 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32903.08 MB 2025-02-14 13:08:28,178 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 13:08:28,178 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 13:08:28,178 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 13:08:28,178 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:08:28,178 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32890.07 MB 2025-02-14 13:08:28,179 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24207.31 MB 2025-02-14 13:08:28,179 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8682.76 MB 2025-02-14 13:08:28,179 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38803.60 MB 2025-02-14 13:08:28,179 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38803.60 MB 2025-02-14 13:08:28,179 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:08:28,179 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35399.90 MB 2025-02-14 13:08:28,197 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8156, cut from 8158 2025-02-14 13:08:28,197 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-14 13:08:28,203 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 13:08:28,203 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 13:08:28,203 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:08:28,203 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:08:28,203 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24207.31 MB 2025-02-14 13:08:28,203 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32640.61 MB 2025-02-14 13:08:28,203 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8433.30 MB 2025-02-14 13:08:28,203 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38803.60 MB 2025-02-14 13:08:28,203 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47188.02 MB 2025-02-14 13:08:28,203 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8384.41 MB 2025-02-14 13:08:28,203 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32640.61 MB 2025-02-14 13:08:28,367 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7948] 2025-02-14 13:08:28,368 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:08:28,368 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 13:08:28,369 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:08:28,369 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 13:08:28,374 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 13:08:28,375 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:08:28,375 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 13:08:28,375 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-14 13:08:55,073 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:08:55,073 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 13:08:55,078 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 13:08:55,082 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:08:55,082 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1614, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 13:08:55,083 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:08:55,083 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1614, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 13:09:20,233 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 13:09:20,233 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 13:09:20,233 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.14 seconds 2025-02-14 13:09:20,233 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:09:20,233 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24215.32 MB 2025-02-14 13:09:20,233 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29927.96 MB 2025-02-14 13:09:20,233 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5712.64 MB 2025-02-14 13:09:20,233 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55572.43 MB 2025-02-14 13:09:20,233 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39382.42 MB 2025-02-14 13:09:20,234 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16190.01 MB 2025-02-14 13:09:20,234 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38896.82 MB 2025-02-14 13:09:20,327 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 13:09:20,327 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 13:09:20,327 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 13:09:20,327 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:09:20,327 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29927.96 MB 2025-02-14 13:09:20,327 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24168.53 MB 2025-02-14 13:09:20,327 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5759.43 MB 2025-02-14 13:09:20,327 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39382.42 MB 2025-02-14 13:09:20,327 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51915.00 MB 2025-02-14 13:09:20,327 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 12532.58 MB 2025-02-14 13:09:20,327 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46321.19 MB 2025-02-14 13:09:22,269 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 13:09:22,269 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 13:09:22,269 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-14 13:09:22,269 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:09:22,269 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24168.53 MB 2025-02-14 13:09:22,269 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24699.37 MB 2025-02-14 13:09:22,269 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 13:09:22,269 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51915.00 MB 2025-02-14 13:09:22,269 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29479.67 MB 2025-02-14 13:09:22,269 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22435.33 MB 2025-02-14 13:09:22,269 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28678.71 MB 2025-02-14 13:09:22,285 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 13:09:22,285 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 13:09:22,285 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:09:22,285 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:09:22,285 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24699.37 MB 2025-02-14 13:09:22,285 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26588.91 MB 2025-02-14 13:09:22,285 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 13:09:22,285 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29479.67 MB 2025-02-14 13:09:22,285 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30423.38 MB 2025-02-14 13:09:22,285 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 13:09:22,285 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28006.34 MB 2025-02-14 13:09:22,500 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 13:09:22,500 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 13:09:22,500 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 13:09:22,500 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:09:22,500 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26588.91 MB 2025-02-14 13:09:22,500 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28830.76 MB 2025-02-14 13:09:22,500 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 13:09:22,500 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30423.38 MB 2025-02-14 13:09:22,500 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36557.55 MB 2025-02-14 13:09:22,500 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 13:09:22,500 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34375.05 MB 2025-02-14 13:09:22,501 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 13:09:22,501 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 13:09:22,501 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 13:09:22,501 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:09:22,501 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24699.37 MB 2025-02-14 13:09:22,501 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28830.76 MB 2025-02-14 13:09:22,501 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 13:09:22,501 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29479.67 MB 2025-02-14 13:09:22,501 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36557.55 MB 2025-02-14 13:09:22,501 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7077.89 MB 2025-02-14 13:09:22,501 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34375.05 MB 2025-02-14 13:09:22,672 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 13:09:22,672 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 13:09:22,672 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 13:09:22,672 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:09:22,672 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30364.31 MB 2025-02-14 13:09:22,672 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31131.31 MB 2025-02-14 13:09:22,672 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 13:09:22,672 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36557.55 MB 2025-02-14 13:09:22,672 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36972.79 MB 2025-02-14 13:09:22,672 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 13:09:22,672 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31839.10 MB 2025-02-14 13:09:22,691 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 13:09:22,691 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 13:09:22,691 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:09:22,691 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:09:22,692 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31544.20 MB 2025-02-14 13:09:22,692 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31773.54 MB 2025-02-14 13:09:22,692 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.35 MB 2025-02-14 13:09:22,692 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36972.79 MB 2025-02-14 13:09:22,692 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36972.79 MB 2025-02-14 13:09:22,692 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:09:22,692 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31988.36 MB 2025-02-14 13:09:22,693 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 13:09:22,693 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 13:09:22,693 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.61 seconds 2025-02-14 13:09:22,693 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:09:22,693 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18592.01 MB 2025-02-14 13:09:22,693 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31974.62 MB 2025-02-14 13:09:22,693 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13382.61 MB 2025-02-14 13:09:22,693 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55572.43 MB 2025-02-14 13:09:22,693 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36972.79 MB 2025-02-14 13:09:22,693 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18599.64 MB 2025-02-14 13:09:22,693 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31988.36 MB 2025-02-14 13:09:22,964 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 13:09:22,964 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 13:09:22,964 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 13:09:22,964 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:09:22,964 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31974.62 MB 2025-02-14 13:09:22,964 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23596.40 MB 2025-02-14 13:09:22,964 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8378.22 MB 2025-02-14 13:09:22,964 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36972.79 MB 2025-02-14 13:09:22,964 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36972.79 MB 2025-02-14 13:09:22,964 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:09:22,965 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34486.28 MB 2025-02-14 13:09:22,982 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 13:09:22,983 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 13:09:22,989 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 13:09:22,989 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 13:09:22,989 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:09:22,989 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:09:22,989 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23596.40 MB 2025-02-14 13:09:22,989 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32035.42 MB 2025-02-14 13:09:22,989 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 13:09:22,989 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36972.79 MB 2025-02-14 13:09:22,989 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45363.49 MB 2025-02-14 13:09:22,989 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 13:09:22,989 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32035.42 MB 2025-02-14 13:09:23,156 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 13:09:23,157 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:09:23,158 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 13:09:23,158 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:09:23,159 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 13:09:23,163 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 13:09:23,164 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:09:23,164 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 13:09:23,165 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 13:10:13,118 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:10:13,118 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 13:10:13,123 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 13:10:13,127 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:10:13,127 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 508, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 13:10:13,128 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:10:13,128 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 508, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 13:10:20,968 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 13:10:20,968 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 13:10:20,968 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.84 seconds 2025-02-14 13:10:20,968 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:10:20,968 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16508.53 MB 2025-02-14 13:10:20,968 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18306.25 MB 2025-02-14 13:10:20,968 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1797.72 MB 2025-02-14 13:10:20,968 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57948.50 MB 2025-02-14 13:10:20,968 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22485.66 MB 2025-02-14 13:10:20,968 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35462.84 MB 2025-02-14 13:10:20,968 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27113.11 MB 2025-02-14 13:10:21,002 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 13:10:21,002 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 13:10:21,002 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 13:10:21,002 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:10:21,003 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18306.25 MB 2025-02-14 13:10:21,003 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18418.78 MB 2025-02-14 13:10:21,003 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 112.53 MB 2025-02-14 13:10:21,003 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22485.66 MB 2025-02-14 13:10:21,003 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28689.04 MB 2025-02-14 13:10:21,003 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6203.38 MB 2025-02-14 13:10:21,003 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25944.62 MB 2025-02-14 13:10:22,925 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 13:10:22,925 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 13:10:22,925 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 13:10:22,925 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:10:22,925 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18418.78 MB 2025-02-14 13:10:22,925 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18949.63 MB 2025-02-14 13:10:22,925 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 13:10:22,925 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28689.04 MB 2025-02-14 13:10:22,925 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21158.17 MB 2025-02-14 13:10:22,925 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7530.87 MB 2025-02-14 13:10:22,925 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22930.00 MB 2025-02-14 13:10:22,939 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 13:10:22,940 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 13:10:22,940 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:10:22,940 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:10:22,940 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18949.63 MB 2025-02-14 13:10:22,940 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20839.16 MB 2025-02-14 13:10:22,940 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 13:10:22,940 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21158.17 MB 2025-02-14 13:10:22,940 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24461.18 MB 2025-02-14 13:10:22,940 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 13:10:22,940 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22256.59 MB 2025-02-14 13:10:23,148 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 13:10:23,148 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 13:10:23,148 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 13:10:23,148 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:10:23,148 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20839.16 MB 2025-02-14 13:10:23,148 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23081.02 MB 2025-02-14 13:10:23,148 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 13:10:23,148 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24461.18 MB 2025-02-14 13:10:23,148 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31067.21 MB 2025-02-14 13:10:23,148 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 13:10:23,148 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28625.30 MB 2025-02-14 13:10:23,149 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 13:10:23,149 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 13:10:23,149 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 13:10:23,149 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:10:23,149 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18949.63 MB 2025-02-14 13:10:23,149 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23081.02 MB 2025-02-14 13:10:23,149 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 13:10:23,149 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21158.17 MB 2025-02-14 13:10:23,149 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31067.21 MB 2025-02-14 13:10:23,149 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9909.04 MB 2025-02-14 13:10:23,149 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28625.30 MB 2025-02-14 13:10:23,310 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 13:10:23,311 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 13:10:23,311 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 13:10:23,311 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:10:23,311 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24614.56 MB 2025-02-14 13:10:23,311 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25381.56 MB 2025-02-14 13:10:23,311 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 13:10:23,311 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31067.21 MB 2025-02-14 13:10:23,311 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31480.35 MB 2025-02-14 13:10:23,311 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 13:10:23,311 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26089.35 MB 2025-02-14 13:10:23,329 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 13:10:23,329 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 13:10:23,329 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:10:23,329 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:10:23,329 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25794.45 MB 2025-02-14 13:10:23,329 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26021.21 MB 2025-02-14 13:10:23,329 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.76 MB 2025-02-14 13:10:23,329 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31480.35 MB 2025-02-14 13:10:23,329 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31480.35 MB 2025-02-14 13:10:23,329 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:10:23,329 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26189.13 MB 2025-02-14 13:10:23,331 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 13:10:23,331 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 13:10:23,331 - resource_logging.py:150 - __exit__ - DEBUG - Time: 10.20 seconds 2025-02-14 13:10:23,331 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:10:23,331 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14738.62 MB 2025-02-14 13:10:23,331 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26222.28 MB 2025-02-14 13:10:23,331 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11483.66 MB 2025-02-14 13:10:23,331 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57948.50 MB 2025-02-14 13:10:23,331 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31480.35 MB 2025-02-14 13:10:23,331 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26468.16 MB 2025-02-14 13:10:23,331 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26222.28 MB 2025-02-14 13:10:23,598 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 13:10:23,598 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 13:10:23,598 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 13:10:23,598 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:10:23,598 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16728.97 MB 2025-02-14 13:10:23,598 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19743.01 MB 2025-02-14 13:10:23,598 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-14 13:10:23,598 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31480.35 MB 2025-02-14 13:10:23,598 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31480.35 MB 2025-02-14 13:10:23,598 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:10:23,598 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20044.38 MB 2025-02-14 13:10:23,616 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 13:10:23,616 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 13:10:23,622 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 13:10:23,622 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 13:10:23,622 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:10:23,622 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:10:23,623 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19743.01 MB 2025-02-14 13:10:23,623 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28182.03 MB 2025-02-14 13:10:23,623 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 13:10:23,623 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31480.35 MB 2025-02-14 13:10:23,623 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41970.30 MB 2025-02-14 13:10:23,623 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 13:10:23,623 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28182.03 MB 2025-02-14 13:10:23,776 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 13:10:23,778 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:10:23,778 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 13:10:23,779 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:10:23,779 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 13:10:23,783 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 13:10:23,784 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:10:23,784 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 13:10:23,784 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 13:11:06,340 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:11:06,341 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 13:11:06,346 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 13:11:06,349 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:11:06,349 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1032, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 13:11:06,350 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:11:06,351 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1032, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 13:11:22,286 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 13:11:22,286 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 13:11:22,286 - resource_logging.py:150 - __exit__ - DEBUG - Time: 15.93 seconds 2025-02-14 13:11:22,286 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:11:22,286 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20159.85 MB 2025-02-14 13:11:22,287 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23813.09 MB 2025-02-14 13:11:22,287 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3653.24 MB 2025-02-14 13:11:22,287 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54555.31 MB 2025-02-14 13:11:22,287 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31048.34 MB 2025-02-14 13:11:22,287 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23506.98 MB 2025-02-14 13:11:22,287 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32802.11 MB 2025-02-14 13:11:22,347 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 13:11:22,347 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 13:11:22,347 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 13:11:22,347 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:11:22,347 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23813.09 MB 2025-02-14 13:11:22,347 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21142.90 MB 2025-02-14 13:11:22,347 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2670.19 MB 2025-02-14 13:11:22,347 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31048.34 MB 2025-02-14 13:11:22,347 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39315.31 MB 2025-02-14 13:11:22,347 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8266.97 MB 2025-02-14 13:11:22,347 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34914.35 MB 2025-02-14 13:11:24,255 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 13:11:24,255 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 13:11:24,255 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 13:11:24,255 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:11:24,256 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21142.90 MB 2025-02-14 13:11:24,256 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21673.74 MB 2025-02-14 13:11:24,256 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 13:11:24,256 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39315.31 MB 2025-02-14 13:11:24,256 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28810.67 MB 2025-02-14 13:11:24,256 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10504.63 MB 2025-02-14 13:11:24,256 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25653.07 MB 2025-02-14 13:11:24,272 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 13:11:24,272 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 13:11:24,272 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:11:24,272 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:11:24,272 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21673.74 MB 2025-02-14 13:11:24,272 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23563.27 MB 2025-02-14 13:11:24,272 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 13:11:24,272 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28810.67 MB 2025-02-14 13:11:24,272 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28810.67 MB 2025-02-14 13:11:24,272 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:11:24,272 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24980.70 MB 2025-02-14 13:11:24,479 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 13:11:24,479 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 13:11:24,479 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 13:11:24,479 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:11:24,479 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23563.27 MB 2025-02-14 13:11:24,479 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25805.13 MB 2025-02-14 13:11:24,479 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 13:11:24,479 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28810.67 MB 2025-02-14 13:11:24,479 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34001.13 MB 2025-02-14 13:11:24,479 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 13:11:24,480 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31349.41 MB 2025-02-14 13:11:24,480 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 13:11:24,480 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 13:11:24,480 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 13:11:24,480 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:11:24,480 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21673.74 MB 2025-02-14 13:11:24,480 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25805.13 MB 2025-02-14 13:11:24,480 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 13:11:24,480 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28810.67 MB 2025-02-14 13:11:24,480 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34001.13 MB 2025-02-14 13:11:24,480 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 13:11:24,480 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31349.41 MB 2025-02-14 13:11:24,643 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 13:11:24,643 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 13:11:24,643 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 13:11:24,643 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:11:24,643 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27338.67 MB 2025-02-14 13:11:24,643 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28105.67 MB 2025-02-14 13:11:24,643 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 13:11:24,643 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34001.13 MB 2025-02-14 13:11:24,643 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34418.46 MB 2025-02-14 13:11:24,643 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 13:11:24,643 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28813.46 MB 2025-02-14 13:11:24,662 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 13:11:24,662 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 13:11:24,662 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:11:24,662 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:11:24,662 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28518.56 MB 2025-02-14 13:11:24,662 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28747.18 MB 2025-02-14 13:11:24,662 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.62 MB 2025-02-14 13:11:24,662 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34418.46 MB 2025-02-14 13:11:24,662 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34418.46 MB 2025-02-14 13:11:24,662 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:11:24,662 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28956.84 MB 2025-02-14 13:11:24,663 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 13:11:24,663 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 13:11:24,663 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.31 seconds 2025-02-14 13:11:24,663 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:11:24,663 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16564.28 MB 2025-02-14 13:11:24,663 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28948.58 MB 2025-02-14 13:11:24,663 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12384.31 MB 2025-02-14 13:11:24,663 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54555.31 MB 2025-02-14 13:11:24,663 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34418.46 MB 2025-02-14 13:11:24,663 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20136.85 MB 2025-02-14 13:11:24,663 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28956.84 MB 2025-02-14 13:11:24,930 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 13:11:24,930 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 13:11:24,930 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 13:11:24,930 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:11:24,930 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28948.58 MB 2025-02-14 13:11:24,930 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21561.16 MB 2025-02-14 13:11:24,930 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7387.42 MB 2025-02-14 13:11:24,930 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34418.46 MB 2025-02-14 13:11:24,930 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34418.46 MB 2025-02-14 13:11:24,930 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:11:24,930 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31453.49 MB 2025-02-14 13:11:24,948 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8140, cut from 8142 2025-02-14 13:11:24,948 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 13:11:24,954 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 13:11:24,954 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 13:11:24,954 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:11:24,954 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:11:24,955 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21561.16 MB 2025-02-14 13:11:24,955 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29977.76 MB 2025-02-14 13:11:24,955 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8416.60 MB 2025-02-14 13:11:24,955 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34418.46 MB 2025-02-14 13:11:24,955 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42786.10 MB 2025-02-14 13:11:24,955 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8367.64 MB 2025-02-14 13:11:24,955 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29977.76 MB 2025-02-14 13:11:25,107 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7932] 2025-02-14 13:11:25,108 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:11:25,108 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 13:11:25,109 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:11:25,109 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 13:11:25,113 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 13:11:25,114 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:11:25,114 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 13:11:25,115 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 13:11:51,826 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:11:51,826 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 13:11:51,830 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 13:11:51,834 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:11:51,834 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1131, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 13:11:51,835 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:11:51,835 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1131, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 13:12:09,408 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 13:12:09,408 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 13:12:09,408 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.57 seconds 2025-02-14 13:12:09,408 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:12:09,408 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20849.70 MB 2025-02-14 13:12:09,408 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24853.16 MB 2025-02-14 13:12:09,408 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4003.46 MB 2025-02-14 13:12:09,408 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51153.73 MB 2025-02-14 13:12:09,408 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31398.56 MB 2025-02-14 13:12:09,408 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19755.17 MB 2025-02-14 13:12:09,408 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33718.45 MB 2025-02-14 13:12:09,482 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 13:12:09,482 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 13:12:09,482 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 13:12:09,482 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:12:09,482 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24853.16 MB 2025-02-14 13:12:09,482 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21657.57 MB 2025-02-14 13:12:09,482 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3195.59 MB 2025-02-14 13:12:09,482 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31398.56 MB 2025-02-14 13:12:09,482 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42234.54 MB 2025-02-14 13:12:09,482 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10835.98 MB 2025-02-14 13:12:09,482 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36955.35 MB 2025-02-14 13:12:11,414 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 13:12:11,414 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 13:12:11,414 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 13:12:11,414 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:12:11,414 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21657.57 MB 2025-02-14 13:12:11,414 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22188.41 MB 2025-02-14 13:12:11,414 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 13:12:11,414 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42234.54 MB 2025-02-14 13:12:11,414 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26711.43 MB 2025-02-14 13:12:11,414 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15523.12 MB 2025-02-14 13:12:11,414 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26168.78 MB 2025-02-14 13:12:11,428 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 13:12:11,428 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 13:12:11,428 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:12:11,428 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:12:11,428 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22188.41 MB 2025-02-14 13:12:11,428 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24077.94 MB 2025-02-14 13:12:11,428 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 13:12:11,428 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26711.43 MB 2025-02-14 13:12:11,428 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28598.86 MB 2025-02-14 13:12:11,428 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 13:12:11,428 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25495.37 MB 2025-02-14 13:12:11,639 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 13:12:11,639 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 13:12:11,639 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 13:12:11,639 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:12:11,639 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24077.94 MB 2025-02-14 13:12:11,639 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26319.80 MB 2025-02-14 13:12:11,639 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 13:12:11,639 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28598.86 MB 2025-02-14 13:12:11,639 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34733.03 MB 2025-02-14 13:12:11,639 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 13:12:11,639 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31864.08 MB 2025-02-14 13:12:11,639 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 13:12:11,639 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 13:12:11,639 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 13:12:11,640 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:12:11,640 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22188.41 MB 2025-02-14 13:12:11,640 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26319.80 MB 2025-02-14 13:12:11,640 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 13:12:11,640 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26711.43 MB 2025-02-14 13:12:11,640 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34733.03 MB 2025-02-14 13:12:11,640 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-14 13:12:11,640 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31864.08 MB 2025-02-14 13:12:11,817 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 13:12:11,817 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 13:12:11,817 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 13:12:11,817 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:12:11,817 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27853.34 MB 2025-02-14 13:12:11,817 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28620.34 MB 2025-02-14 13:12:11,817 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 13:12:11,817 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34733.03 MB 2025-02-14 13:12:11,817 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35150.36 MB 2025-02-14 13:12:11,817 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 13:12:11,817 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29328.13 MB 2025-02-14 13:12:11,836 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 13:12:11,836 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 13:12:11,836 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:12:11,836 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:12:11,836 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29033.23 MB 2025-02-14 13:12:11,836 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29262.54 MB 2025-02-14 13:12:11,836 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.31 MB 2025-02-14 13:12:11,836 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35150.36 MB 2025-02-14 13:12:11,836 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35150.36 MB 2025-02-14 13:12:11,836 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:12:11,836 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29472.88 MB 2025-02-14 13:12:11,837 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 13:12:11,837 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 13:12:11,837 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.00 seconds 2025-02-14 13:12:11,837 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:12:11,837 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16909.20 MB 2025-02-14 13:12:11,837 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29463.61 MB 2025-02-14 13:12:11,837 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12554.41 MB 2025-02-14 13:12:11,837 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51153.73 MB 2025-02-14 13:12:11,837 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35150.36 MB 2025-02-14 13:12:11,837 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16003.37 MB 2025-02-14 13:12:11,837 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29472.88 MB 2025-02-14 13:12:12,107 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 13:12:12,107 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 13:12:12,107 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 13:12:12,107 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:12:12,107 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29463.61 MB 2025-02-14 13:12:12,107 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21913.59 MB 2025-02-14 13:12:12,107 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7550.02 MB 2025-02-14 13:12:12,107 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35150.36 MB 2025-02-14 13:12:12,107 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35150.36 MB 2025-02-14 13:12:12,107 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:12:12,107 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31975.28 MB 2025-02-14 13:12:12,125 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 13:12:12,125 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-14 13:12:12,131 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 13:12:12,131 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 13:12:12,131 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:12:12,131 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:12:12,131 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21913.59 MB 2025-02-14 13:12:12,131 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30352.61 MB 2025-02-14 13:12:12,131 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 13:12:12,131 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35150.36 MB 2025-02-14 13:12:12,131 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43541.07 MB 2025-02-14 13:12:12,131 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 13:12:12,131 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30352.61 MB 2025-02-14 13:12:12,295 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 13:12:12,296 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:12:12,296 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 13:12:12,297 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:12:12,297 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 13:12:12,302 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 13:12:12,303 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:12:12,303 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 13:12:12,303 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-14 13:13:14,346 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:13:14,346 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 13:13:14,351 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 13:13:14,355 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:13:14,355 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 506, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 13:13:14,356 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:13:14,356 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 506, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 13:13:22,159 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 13:13:22,159 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 13:13:22,159 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.80 seconds 2025-02-14 13:13:22,159 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:13:22,159 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16494.60 MB 2025-02-14 13:13:22,159 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18285.56 MB 2025-02-14 13:13:22,159 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1790.97 MB 2025-02-14 13:13:22,159 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56126.08 MB 2025-02-14 13:13:22,159 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22475.18 MB 2025-02-14 13:13:22,159 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33650.90 MB 2025-02-14 13:13:22,159 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27099.23 MB 2025-02-14 13:13:22,196 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 13:13:22,196 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 13:13:22,196 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 13:13:22,196 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:13:22,196 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18285.56 MB 2025-02-14 13:13:22,196 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18408.39 MB 2025-02-14 13:13:22,196 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 122.82 MB 2025-02-14 13:13:22,196 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22475.18 MB 2025-02-14 13:13:22,196 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28800.19 MB 2025-02-14 13:13:22,196 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6325.01 MB 2025-02-14 13:13:22,196 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26036.84 MB 2025-02-14 13:13:24,159 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 13:13:24,159 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 13:13:24,159 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.96 seconds 2025-02-14 13:13:24,159 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:13:24,159 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18408.39 MB 2025-02-14 13:13:24,160 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18939.23 MB 2025-02-14 13:13:24,160 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 13:13:24,160 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28800.19 MB 2025-02-14 13:13:24,160 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21156.07 MB 2025-02-14 13:13:24,160 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7644.12 MB 2025-02-14 13:13:24,160 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22919.60 MB 2025-02-14 13:13:24,174 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 13:13:24,174 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 13:13:24,174 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:13:24,174 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:13:24,174 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18939.23 MB 2025-02-14 13:13:24,174 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20828.76 MB 2025-02-14 13:13:24,174 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 13:13:24,174 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21156.07 MB 2025-02-14 13:13:24,174 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24459.08 MB 2025-02-14 13:13:24,174 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 13:13:24,174 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22246.19 MB 2025-02-14 13:13:24,388 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 13:13:24,388 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 13:13:24,388 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 13:13:24,388 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:13:24,388 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20828.76 MB 2025-02-14 13:13:24,388 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23070.62 MB 2025-02-14 13:13:24,388 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 13:13:24,388 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24459.08 MB 2025-02-14 13:13:24,388 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31065.11 MB 2025-02-14 13:13:24,388 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 13:13:24,388 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28614.90 MB 2025-02-14 13:13:24,388 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 13:13:24,388 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 13:13:24,389 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 13:13:24,389 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:13:24,389 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18939.23 MB 2025-02-14 13:13:24,389 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23070.62 MB 2025-02-14 13:13:24,389 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 13:13:24,389 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21156.07 MB 2025-02-14 13:13:24,389 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31065.11 MB 2025-02-14 13:13:24,389 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9909.04 MB 2025-02-14 13:13:24,389 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28614.90 MB 2025-02-14 13:13:24,554 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 13:13:24,554 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 13:13:24,554 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 13:13:24,554 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:13:24,554 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24604.16 MB 2025-02-14 13:13:24,554 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25371.16 MB 2025-02-14 13:13:24,554 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 13:13:24,554 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31065.11 MB 2025-02-14 13:13:24,554 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31480.35 MB 2025-02-14 13:13:24,554 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 13:13:24,554 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26078.95 MB 2025-02-14 13:13:24,573 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 13:13:24,573 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 13:13:24,574 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:13:24,574 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:13:24,574 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25784.05 MB 2025-02-14 13:13:24,574 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26011.86 MB 2025-02-14 13:13:24,574 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.81 MB 2025-02-14 13:13:24,574 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31480.35 MB 2025-02-14 13:13:24,574 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31480.35 MB 2025-02-14 13:13:24,574 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:13:24,574 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26232.68 MB 2025-02-14 13:13:24,575 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 13:13:24,575 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 13:13:24,575 - resource_logging.py:150 - __exit__ - DEBUG - Time: 10.22 seconds 2025-02-14 13:13:24,575 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:13:24,575 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14731.65 MB 2025-02-14 13:13:24,575 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26212.68 MB 2025-02-14 13:13:24,575 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11481.03 MB 2025-02-14 13:13:24,575 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56126.08 MB 2025-02-14 13:13:24,575 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31480.35 MB 2025-02-14 13:13:24,575 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24645.73 MB 2025-02-14 13:13:24,575 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26232.68 MB 2025-02-14 13:13:24,842 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 13:13:24,842 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 13:13:24,842 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 13:13:24,842 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:13:24,842 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26212.68 MB 2025-02-14 13:13:24,842 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19732.23 MB 2025-02-14 13:13:24,842 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6480.45 MB 2025-02-14 13:13:24,842 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31480.35 MB 2025-02-14 13:13:24,842 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31480.35 MB 2025-02-14 13:13:24,842 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:13:24,842 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28721.28 MB 2025-02-14 13:13:24,860 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8152, cut from 8154 2025-02-14 13:13:24,860 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-14 13:13:24,866 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 13:13:24,866 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 13:13:24,866 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:13:24,866 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:13:24,866 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19732.23 MB 2025-02-14 13:13:24,866 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28161.35 MB 2025-02-14 13:13:24,866 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8429.12 MB 2025-02-14 13:13:24,866 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31480.35 MB 2025-02-14 13:13:24,866 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41955.62 MB 2025-02-14 13:13:24,866 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10475.27 MB 2025-02-14 13:13:24,866 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28161.35 MB 2025-02-14 13:13:25,020 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7944] 2025-02-14 13:13:25,021 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:13:25,022 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 13:13:25,022 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:13:25,023 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 13:13:25,027 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 13:13:25,028 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:13:25,028 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 13:13:25,028 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-14 13:15:47,095 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:15:47,095 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 13:15:47,100 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 13:15:47,104 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:15:47,104 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1334, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 13:15:47,105 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:15:47,105 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1334, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 13:16:07,511 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 13:16:07,512 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 13:16:07,512 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.40 seconds 2025-02-14 13:16:07,512 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:16:07,512 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22264.23 MB 2025-02-14 13:16:07,512 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26985.18 MB 2025-02-14 13:16:07,512 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4720.95 MB 2025-02-14 13:16:07,512 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50335.84 MB 2025-02-14 13:16:07,512 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38390.46 MB 2025-02-14 13:16:07,512 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11945.38 MB 2025-02-14 13:16:07,512 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35812.47 MB 2025-02-14 13:16:07,614 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 13:16:07,614 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 13:16:07,614 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 13:16:07,614 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:16:07,614 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26985.18 MB 2025-02-14 13:16:07,614 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22712.90 MB 2025-02-14 13:16:07,614 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4272.28 MB 2025-02-14 13:16:07,614 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38390.46 MB 2025-02-14 13:16:07,614 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47726.99 MB 2025-02-14 13:16:07,614 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9336.52 MB 2025-02-14 13:16:07,614 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41045.62 MB 2025-02-14 13:16:09,518 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 13:16:09,518 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 13:16:09,518 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.90 seconds 2025-02-14 13:16:09,518 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:16:09,518 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22712.90 MB 2025-02-14 13:16:09,518 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23243.74 MB 2025-02-14 13:16:09,518 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 13:16:09,518 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47726.99 MB 2025-02-14 13:16:09,518 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33667.68 MB 2025-02-14 13:16:09,518 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14059.31 MB 2025-02-14 13:16:09,518 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27223.08 MB 2025-02-14 13:16:09,531 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 13:16:09,532 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 13:16:09,532 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:16:09,532 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:16:09,532 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23243.74 MB 2025-02-14 13:16:09,532 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25133.28 MB 2025-02-14 13:16:09,532 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 13:16:09,532 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33667.68 MB 2025-02-14 13:16:09,532 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33667.68 MB 2025-02-14 13:16:09,532 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:16:09,532 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26550.71 MB 2025-02-14 13:16:09,744 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 13:16:09,745 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 13:16:09,745 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 13:16:09,745 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:16:09,745 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25133.28 MB 2025-02-14 13:16:09,745 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27375.13 MB 2025-02-14 13:16:09,745 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 13:16:09,745 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33667.68 MB 2025-02-14 13:16:09,745 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35555.12 MB 2025-02-14 13:16:09,745 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 13:16:09,745 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32919.41 MB 2025-02-14 13:16:09,745 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 13:16:09,745 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 13:16:09,745 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 13:16:09,745 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:16:09,745 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23243.74 MB 2025-02-14 13:16:09,745 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27375.13 MB 2025-02-14 13:16:09,745 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 13:16:09,745 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33667.68 MB 2025-02-14 13:16:09,745 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35555.12 MB 2025-02-14 13:16:09,745 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 13:16:09,745 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32919.41 MB 2025-02-14 13:16:09,924 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 13:16:09,924 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 13:16:09,924 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 13:16:09,924 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:16:09,924 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28908.67 MB 2025-02-14 13:16:09,924 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29675.68 MB 2025-02-14 13:16:09,924 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 13:16:09,924 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35555.12 MB 2025-02-14 13:16:09,924 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35972.45 MB 2025-02-14 13:16:09,924 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 13:16:09,924 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30383.46 MB 2025-02-14 13:16:09,944 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 13:16:09,944 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 13:16:09,945 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:16:09,945 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:16:09,945 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30088.57 MB 2025-02-14 13:16:09,945 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30316.96 MB 2025-02-14 13:16:09,945 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.40 MB 2025-02-14 13:16:09,945 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35972.45 MB 2025-02-14 13:16:09,945 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35972.45 MB 2025-02-14 13:16:09,945 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:16:09,945 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30544.82 MB 2025-02-14 13:16:09,946 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 13:16:09,946 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 13:16:09,946 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.84 seconds 2025-02-14 13:16:09,946 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:16:09,946 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17616.47 MB 2025-02-14 13:16:09,946 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30517.27 MB 2025-02-14 13:16:09,946 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12900.80 MB 2025-02-14 13:16:09,946 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50335.84 MB 2025-02-14 13:16:09,946 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35972.45 MB 2025-02-14 13:16:09,946 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14363.39 MB 2025-02-14 13:16:09,946 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30544.82 MB 2025-02-14 13:16:10,213 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 13:16:10,213 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 13:16:10,213 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 13:16:10,213 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:16:10,213 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30517.27 MB 2025-02-14 13:16:10,213 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22609.05 MB 2025-02-14 13:16:10,213 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7908.22 MB 2025-02-14 13:16:10,213 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35972.45 MB 2025-02-14 13:16:10,213 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35972.45 MB 2025-02-14 13:16:10,213 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:16:10,213 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33019.42 MB 2025-02-14 13:16:10,231 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8131, cut from 8133 2025-02-14 13:16:10,231 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 13:16:10,237 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 13:16:10,237 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 13:16:10,237 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:16:10,237 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:16:10,237 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22609.05 MB 2025-02-14 13:16:10,237 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31015.74 MB 2025-02-14 13:16:10,237 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8406.69 MB 2025-02-14 13:16:10,237 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35972.45 MB 2025-02-14 13:16:10,237 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40152.07 MB 2025-02-14 13:16:10,237 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4179.62 MB 2025-02-14 13:16:10,237 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31015.74 MB 2025-02-14 13:16:10,400 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7923] 2025-02-14 13:16:10,401 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:16:10,401 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 13:16:10,402 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:16:10,402 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 13:16:10,407 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 13:16:10,408 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:16:10,408 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 13:16:10,408 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 13:17:04,905 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:17:04,905 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 13:17:04,910 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 13:17:04,914 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:17:04,914 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 3345, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 13:17:04,915 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:17:04,915 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 3345, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 13:17:56,838 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 13:17:56,838 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 13:17:56,838 - resource_logging.py:150 - __exit__ - DEBUG - Time: 51.91 seconds 2025-02-14 13:17:56,838 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:17:56,838 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36277.90 MB 2025-02-14 13:17:56,838 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48116.32 MB 2025-02-14 13:17:56,838 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11838.42 MB 2025-02-14 13:17:56,838 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 71823.26 MB 2025-02-14 13:17:56,838 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52059.70 MB 2025-02-14 13:17:56,838 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19763.56 MB 2025-02-14 13:17:56,838 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 59954.74 MB 2025-02-14 13:17:57,078 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 13:17:57,078 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 13:17:57,078 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.24 seconds 2025-02-14 13:17:57,078 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:17:57,078 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 48116.32 MB 2025-02-14 13:17:57,078 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33168.16 MB 2025-02-14 13:17:57,078 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -14948.16 MB 2025-02-14 13:17:57,078 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52059.70 MB 2025-02-14 13:17:57,078 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 97282.69 MB 2025-02-14 13:17:57,078 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 45222.99 MB 2025-02-14 13:17:57,078 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 82366.15 MB 2025-02-14 13:17:59,072 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 13:17:59,072 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 13:17:59,072 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.99 seconds 2025-02-14 13:17:59,072 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:17:59,072 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33168.16 MB 2025-02-14 13:17:59,072 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33699.00 MB 2025-02-14 13:17:59,072 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 13:17:59,072 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 97282.69 MB 2025-02-14 13:17:59,072 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35716.60 MB 2025-02-14 13:17:59,072 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -61566.09 MB 2025-02-14 13:17:59,072 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37679.37 MB 2025-02-14 13:17:59,086 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 13:17:59,086 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 13:17:59,086 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:17:59,086 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:17:59,086 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33699.00 MB 2025-02-14 13:17:59,086 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35588.53 MB 2025-02-14 13:17:59,086 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 13:17:59,086 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35716.60 MB 2025-02-14 13:17:59,086 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39019.61 MB 2025-02-14 13:17:59,086 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 13:17:59,086 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37005.96 MB 2025-02-14 13:17:59,298 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 13:17:59,298 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 13:17:59,298 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 13:17:59,299 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:17:59,299 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35588.53 MB 2025-02-14 13:17:59,299 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37830.39 MB 2025-02-14 13:17:59,299 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 13:17:59,299 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39019.61 MB 2025-02-14 13:17:59,299 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45625.64 MB 2025-02-14 13:17:59,299 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 13:17:59,299 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43374.67 MB 2025-02-14 13:17:59,299 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 13:17:59,299 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 13:17:59,299 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 13:17:59,299 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:17:59,299 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33699.00 MB 2025-02-14 13:17:59,299 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37830.39 MB 2025-02-14 13:17:59,299 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 13:17:59,299 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35716.60 MB 2025-02-14 13:17:59,299 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45625.64 MB 2025-02-14 13:17:59,299 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9909.04 MB 2025-02-14 13:17:59,299 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43374.67 MB 2025-02-14 13:17:59,472 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 13:17:59,472 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 13:17:59,472 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 13:17:59,472 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:17:59,472 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39363.93 MB 2025-02-14 13:17:59,472 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40130.93 MB 2025-02-14 13:17:59,472 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 13:17:59,472 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45625.64 MB 2025-02-14 13:17:59,472 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46040.88 MB 2025-02-14 13:17:59,472 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 13:17:59,472 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40838.72 MB 2025-02-14 13:17:59,492 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 13:17:59,492 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 13:17:59,492 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:17:59,492 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:17:59,492 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40543.82 MB 2025-02-14 13:17:59,492 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40771.85 MB 2025-02-14 13:17:59,492 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.03 MB 2025-02-14 13:17:59,492 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46040.88 MB 2025-02-14 13:17:59,492 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46040.88 MB 2025-02-14 13:17:59,492 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:17:59,492 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40992.50 MB 2025-02-14 13:17:59,493 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 13:17:59,493 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 13:17:59,493 - resource_logging.py:150 - __exit__ - DEBUG - Time: 54.58 seconds 2025-02-14 13:17:59,493 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:17:59,493 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24623.30 MB 2025-02-14 13:17:59,493 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40971.81 MB 2025-02-14 13:17:59,493 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 16348.51 MB 2025-02-14 13:17:59,493 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60167.29 MB 2025-02-14 13:17:59,493 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46040.88 MB 2025-02-14 13:17:59,493 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14126.42 MB 2025-02-14 13:17:59,493 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40992.50 MB 2025-02-14 13:17:59,766 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 13:17:59,766 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 13:17:59,766 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 13:17:59,766 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:17:59,766 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40971.81 MB 2025-02-14 13:17:59,766 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29610.55 MB 2025-02-14 13:17:59,766 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -11361.27 MB 2025-02-14 13:17:59,766 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46040.88 MB 2025-02-14 13:17:59,766 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46040.88 MB 2025-02-14 13:17:59,766 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:17:59,766 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43469.66 MB 2025-02-14 13:17:59,784 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8117, cut from 8119 2025-02-14 13:17:59,784 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 13:17:59,790 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 13:17:59,790 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 13:17:59,790 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:17:59,790 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:17:59,790 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29610.55 MB 2025-02-14 13:17:59,790 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38002.79 MB 2025-02-14 13:17:59,790 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8392.24 MB 2025-02-14 13:17:59,790 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46040.88 MB 2025-02-14 13:17:59,790 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50214.21 MB 2025-02-14 13:17:59,790 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4173.33 MB 2025-02-14 13:17:59,790 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38002.79 MB 2025-02-14 13:17:59,952 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7909] 2025-02-14 13:17:59,953 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:17:59,953 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 13:17:59,954 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:17:59,954 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 13:17:59,959 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 13:17:59,960 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:17:59,960 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 13:17:59,960 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 13:19:35,993 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:19:35,993 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 13:19:35,998 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 13:19:36,002 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:19:36,002 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1669, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 13:19:36,003 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:19:36,003 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1669, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 13:20:01,756 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 13:20:01,757 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 13:20:01,757 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.75 seconds 2025-02-14 13:20:01,757 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:20:01,757 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24598.57 MB 2025-02-14 13:20:01,757 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30505.06 MB 2025-02-14 13:20:01,757 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5906.50 MB 2025-02-14 13:20:01,757 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58556.68 MB 2025-02-14 13:20:01,757 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41068.53 MB 2025-02-14 13:20:01,757 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17488.15 MB 2025-02-14 13:20:01,757 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39505.75 MB 2025-02-14 13:20:01,828 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 13:20:01,828 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 13:20:01,828 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 13:20:01,828 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:20:01,828 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30505.06 MB 2025-02-14 13:20:01,828 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24454.46 MB 2025-02-14 13:20:01,829 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6050.60 MB 2025-02-14 13:20:01,829 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41068.53 MB 2025-02-14 13:20:01,829 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45032.14 MB 2025-02-14 13:20:01,829 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3963.62 MB 2025-02-14 13:20:01,829 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39804.12 MB 2025-02-14 13:20:03,769 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 13:20:03,770 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 13:20:03,770 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-14 13:20:03,770 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:20:03,770 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24454.46 MB 2025-02-14 13:20:03,770 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24985.30 MB 2025-02-14 13:20:03,770 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 13:20:03,770 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45032.14 MB 2025-02-14 13:20:03,770 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32403.10 MB 2025-02-14 13:20:03,770 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12629.05 MB 2025-02-14 13:20:03,770 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28964.64 MB 2025-02-14 13:20:03,783 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 13:20:03,784 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 13:20:03,784 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:20:03,784 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:20:03,784 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24985.30 MB 2025-02-14 13:20:03,784 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26874.84 MB 2025-02-14 13:20:03,784 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 13:20:03,784 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32403.10 MB 2025-02-14 13:20:03,784 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32403.10 MB 2025-02-14 13:20:03,784 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:20:03,784 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28292.27 MB 2025-02-14 13:20:04,003 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 13:20:04,003 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 13:20:04,003 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 13:20:04,003 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:20:04,003 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26874.84 MB 2025-02-14 13:20:04,003 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29116.69 MB 2025-02-14 13:20:04,003 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 13:20:04,003 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32403.10 MB 2025-02-14 13:20:04,003 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37121.69 MB 2025-02-14 13:20:04,003 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 13:20:04,003 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34660.97 MB 2025-02-14 13:20:04,004 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 13:20:04,004 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 13:20:04,004 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 13:20:04,004 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:20:04,004 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24985.30 MB 2025-02-14 13:20:04,004 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29116.69 MB 2025-02-14 13:20:04,004 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 13:20:04,004 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32403.10 MB 2025-02-14 13:20:04,004 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37121.69 MB 2025-02-14 13:20:04,004 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 13:20:04,004 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34660.97 MB 2025-02-14 13:20:04,176 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 13:20:04,177 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 13:20:04,177 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 13:20:04,177 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:20:04,177 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30650.24 MB 2025-02-14 13:20:04,177 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31417.24 MB 2025-02-14 13:20:04,177 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 13:20:04,177 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37121.69 MB 2025-02-14 13:20:04,177 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37536.92 MB 2025-02-14 13:20:04,177 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 13:20:04,177 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32125.03 MB 2025-02-14 13:20:04,196 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 13:20:04,196 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 13:20:04,196 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:20:04,196 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:20:04,196 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31830.13 MB 2025-02-14 13:20:04,196 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32059.91 MB 2025-02-14 13:20:04,196 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.78 MB 2025-02-14 13:20:04,196 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37536.92 MB 2025-02-14 13:20:04,196 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37536.92 MB 2025-02-14 13:20:04,196 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:20:04,196 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32254.57 MB 2025-02-14 13:20:04,197 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 13:20:04,197 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 13:20:04,197 - resource_logging.py:150 - __exit__ - DEBUG - Time: 28.19 seconds 2025-02-14 13:20:04,197 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:20:04,197 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18783.64 MB 2025-02-14 13:20:04,197 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32260.98 MB 2025-02-14 13:20:04,197 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13477.35 MB 2025-02-14 13:20:04,197 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58556.68 MB 2025-02-14 13:20:04,197 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37536.92 MB 2025-02-14 13:20:04,198 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21019.75 MB 2025-02-14 13:20:04,198 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32260.98 MB 2025-02-14 13:20:04,469 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 13:20:04,469 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 13:20:04,469 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 13:20:04,469 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:20:04,469 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32260.98 MB 2025-02-14 13:20:04,469 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23788.03 MB 2025-02-14 13:20:04,469 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8472.96 MB 2025-02-14 13:20:04,469 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37536.92 MB 2025-02-14 13:20:04,469 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37536.92 MB 2025-02-14 13:20:04,469 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:20:04,469 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34772.65 MB 2025-02-14 13:20:04,487 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 13:20:04,488 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 13:20:04,494 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 13:20:04,494 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 13:20:04,494 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:20:04,494 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:20:04,494 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23788.03 MB 2025-02-14 13:20:04,494 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32226.72 MB 2025-02-14 13:20:04,494 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8438.69 MB 2025-02-14 13:20:04,494 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37536.92 MB 2025-02-14 13:20:04,494 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41733.32 MB 2025-02-14 13:20:04,494 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4196.40 MB 2025-02-14 13:20:04,494 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32226.72 MB 2025-02-14 13:20:04,654 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 13:20:04,655 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:20:04,655 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 13:20:04,656 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:20:04,656 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 13:20:04,661 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 13:20:04,662 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:20:04,662 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 13:20:04,662 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 13:20:28,086 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:20:28,086 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 13:20:28,094 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 13:20:28,102 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:20:28,102 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1968, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 13:20:28,104 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:20:28,104 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1968, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 13:20:58,873 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 13:20:58,874 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 13:20:58,874 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.76 seconds 2025-02-14 13:20:58,874 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:20:58,874 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26682.05 MB 2025-02-14 13:20:58,874 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33646.69 MB 2025-02-14 13:20:58,874 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6964.64 MB 2025-02-14 13:20:58,874 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50121.93 MB 2025-02-14 13:20:58,874 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40607.15 MB 2025-02-14 13:20:58,874 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9514.78 MB 2025-02-14 13:20:58,874 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42496.01 MB 2025-02-14 13:20:58,996 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 13:20:58,996 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 13:20:58,997 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 13:20:58,997 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:20:58,997 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33646.69 MB 2025-02-14 13:20:58,997 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26008.87 MB 2025-02-14 13:20:58,997 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7637.82 MB 2025-02-14 13:20:58,997 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40607.15 MB 2025-02-14 13:20:58,997 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 63260.59 MB 2025-02-14 13:20:58,997 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 22653.44 MB 2025-02-14 13:20:58,997 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 53583.26 MB 2025-02-14 13:21:00,961 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 13:21:00,961 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 13:21:00,961 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.96 seconds 2025-02-14 13:21:00,961 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:21:00,961 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26008.87 MB 2025-02-14 13:21:00,961 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26539.71 MB 2025-02-14 13:21:00,961 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 13:21:00,961 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63260.59 MB 2025-02-14 13:21:00,961 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30886.85 MB 2025-02-14 13:21:00,961 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32373.74 MB 2025-02-14 13:21:00,961 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30520.08 MB 2025-02-14 13:21:00,975 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 13:21:00,975 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 13:21:00,975 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:21:00,975 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:21:00,975 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26539.71 MB 2025-02-14 13:21:00,975 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28429.24 MB 2025-02-14 13:21:00,975 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 13:21:00,975 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30886.85 MB 2025-02-14 13:21:00,975 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32774.29 MB 2025-02-14 13:21:00,975 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 13:21:00,975 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29846.67 MB 2025-02-14 13:21:01,187 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 13:21:01,187 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 13:21:01,187 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 13:21:01,187 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:21:01,187 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28429.24 MB 2025-02-14 13:21:01,187 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30671.10 MB 2025-02-14 13:21:01,187 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 13:21:01,187 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32774.29 MB 2025-02-14 13:21:01,187 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38908.46 MB 2025-02-14 13:21:01,187 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 13:21:01,187 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36215.38 MB 2025-02-14 13:21:01,188 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 13:21:01,188 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 13:21:01,188 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 13:21:01,188 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:21:01,188 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26539.71 MB 2025-02-14 13:21:01,188 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30671.10 MB 2025-02-14 13:21:01,188 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 13:21:01,188 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30886.85 MB 2025-02-14 13:21:01,188 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38908.46 MB 2025-02-14 13:21:01,188 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-14 13:21:01,188 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36215.38 MB 2025-02-14 13:21:01,358 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 13:21:01,358 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 13:21:01,358 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 13:21:01,358 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:21:01,358 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32204.64 MB 2025-02-14 13:21:01,358 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32971.64 MB 2025-02-14 13:21:01,358 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 13:21:01,358 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38908.46 MB 2025-02-14 13:21:01,358 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39321.60 MB 2025-02-14 13:21:01,358 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 13:21:01,358 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33679.43 MB 2025-02-14 13:21:01,377 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 13:21:01,377 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 13:21:01,377 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:21:01,377 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:21:01,377 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33384.53 MB 2025-02-14 13:21:01,377 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33613.76 MB 2025-02-14 13:21:01,377 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.23 MB 2025-02-14 13:21:01,377 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39321.60 MB 2025-02-14 13:21:01,377 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39321.60 MB 2025-02-14 13:21:01,377 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:21:01,377 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33843.47 MB 2025-02-14 13:21:01,378 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 13:21:01,378 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 13:21:01,378 - resource_logging.py:150 - __exit__ - DEBUG - Time: 33.27 seconds 2025-02-14 13:21:01,378 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:21:01,378 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19825.38 MB 2025-02-14 13:21:01,378 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33814.17 MB 2025-02-14 13:21:01,378 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13988.80 MB 2025-02-14 13:21:01,378 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50121.93 MB 2025-02-14 13:21:01,378 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39321.60 MB 2025-02-14 13:21:01,378 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10800.33 MB 2025-02-14 13:21:01,378 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33843.47 MB 2025-02-14 13:21:01,648 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 13:21:01,648 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 13:21:01,648 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 13:21:01,648 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:21:01,648 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33814.17 MB 2025-02-14 13:21:01,648 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24819.48 MB 2025-02-14 13:21:01,648 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8994.69 MB 2025-02-14 13:21:01,648 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39321.60 MB 2025-02-14 13:21:01,648 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39321.60 MB 2025-02-14 13:21:01,648 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:21:01,648 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36317.55 MB 2025-02-14 13:21:01,666 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8135, cut from 8137 2025-02-14 13:21:01,667 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 13:21:01,673 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 13:21:01,673 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 13:21:01,673 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:21:01,673 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:21:01,673 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24819.48 MB 2025-02-14 13:21:01,673 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33230.30 MB 2025-02-14 13:21:01,673 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8410.82 MB 2025-02-14 13:21:01,673 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39321.60 MB 2025-02-14 13:21:01,673 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47685.04 MB 2025-02-14 13:21:01,673 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8363.44 MB 2025-02-14 13:21:01,673 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33230.30 MB 2025-02-14 13:21:01,835 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7927] 2025-02-14 13:21:01,837 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:21:01,837 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 13:21:01,838 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:21:01,838 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 13:21:01,843 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 13:21:01,844 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:21:01,844 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 13:21:01,844 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 13:22:49,717 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:22:49,717 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 13:22:49,722 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 13:22:49,726 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:22:49,726 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 311, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 13:22:49,727 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:22:49,727 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 311, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 13:22:54,511 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 13:22:54,511 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 13:22:54,511 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.78 seconds 2025-02-14 13:22:54,511 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:22:54,511 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15135.80 MB 2025-02-14 13:22:54,511 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16236.81 MB 2025-02-14 13:22:54,511 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1101.00 MB 2025-02-14 13:22:54,511 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56048.48 MB 2025-02-14 13:22:54,511 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19899.88 MB 2025-02-14 13:22:54,511 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -36148.61 MB 2025-02-14 13:22:54,511 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25060.97 MB 2025-02-14 13:22:54,532 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 13:22:54,532 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 13:22:54,532 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:22:54,532 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:22:54,532 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16236.81 MB 2025-02-14 13:22:54,532 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16763.49 MB 2025-02-14 13:22:54,532 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 526.68 MB 2025-02-14 13:22:54,532 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19899.88 MB 2025-02-14 13:22:54,532 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23179.82 MB 2025-02-14 13:22:54,532 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3279.95 MB 2025-02-14 13:22:54,532 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20616.77 MB 2025-02-14 13:22:56,026 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 13:22:56,026 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 13:22:56,026 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.49 seconds 2025-02-14 13:22:56,026 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:22:56,027 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16763.49 MB 2025-02-14 13:22:56,027 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17174.89 MB 2025-02-14 13:22:56,027 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 411.40 MB 2025-02-14 13:22:56,027 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23179.82 MB 2025-02-14 13:22:56,027 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20036.19 MB 2025-02-14 13:22:56,027 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3143.63 MB 2025-02-14 13:22:56,027 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21104.83 MB 2025-02-14 13:22:56,041 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 13:22:56,041 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 13:22:56,041 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:22:56,041 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:22:56,041 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17174.89 MB 2025-02-14 13:22:56,041 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18640.54 MB 2025-02-14 13:22:56,041 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1465.65 MB 2025-02-14 13:22:56,041 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20036.19 MB 2025-02-14 13:22:56,041 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21500.00 MB 2025-02-14 13:22:56,041 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1463.81 MB 2025-02-14 13:22:56,041 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19739.05 MB 2025-02-14 13:22:56,249 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 13:22:56,249 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 13:22:56,249 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 13:22:56,249 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:22:56,249 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18640.54 MB 2025-02-14 13:22:56,249 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20378.51 MB 2025-02-14 13:22:56,249 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1737.98 MB 2025-02-14 13:22:56,249 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21500.00 MB 2025-02-14 13:22:56,249 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26258.44 MB 2025-02-14 13:22:56,249 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4758.44 MB 2025-02-14 13:22:56,249 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24677.42 MB 2025-02-14 13:22:56,251 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 13:22:56,251 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 13:22:56,251 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 13:22:56,251 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:22:56,251 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17174.89 MB 2025-02-14 13:22:56,251 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20378.51 MB 2025-02-14 13:22:56,251 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3203.62 MB 2025-02-14 13:22:56,251 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20036.19 MB 2025-02-14 13:22:56,251 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26258.44 MB 2025-02-14 13:22:56,251 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6222.25 MB 2025-02-14 13:22:56,251 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24677.42 MB 2025-02-14 13:22:56,457 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 13:22:56,457 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 13:22:56,457 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 13:22:56,457 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:22:56,457 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21567.01 MB 2025-02-14 13:22:56,457 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22161.44 MB 2025-02-14 13:22:56,457 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 594.43 MB 2025-02-14 13:22:56,457 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26258.44 MB 2025-02-14 13:22:56,457 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26577.21 MB 2025-02-14 13:22:56,457 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 318.77 MB 2025-02-14 13:22:56,457 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22709.97 MB 2025-02-14 13:22:56,480 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 13:22:56,480 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 13:22:56,480 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:22:56,480 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:22:56,480 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22481.43 MB 2025-02-14 13:22:56,480 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22701.67 MB 2025-02-14 13:22:56,480 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 220.24 MB 2025-02-14 13:22:56,480 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26577.21 MB 2025-02-14 13:22:56,480 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26577.21 MB 2025-02-14 13:22:56,480 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:22:56,480 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22846.37 MB 2025-02-14 13:22:56,482 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 13:22:56,482 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 13:22:56,482 - resource_logging.py:150 - __exit__ - DEBUG - Time: 6.75 seconds 2025-02-14 13:22:56,482 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:22:56,482 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14052.26 MB 2025-02-14 13:22:56,482 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22902.62 MB 2025-02-14 13:22:56,482 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8850.36 MB 2025-02-14 13:22:56,482 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56048.48 MB 2025-02-14 13:22:56,482 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26577.21 MB 2025-02-14 13:22:56,482 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29471.28 MB 2025-02-14 13:22:56,482 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22902.62 MB 2025-02-14 13:22:56,758 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 13:22:56,758 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 13:22:56,758 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 13:22:56,758 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:22:56,758 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22902.62 MB 2025-02-14 13:22:56,758 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25914.81 MB 2025-02-14 13:22:56,758 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3012.19 MB 2025-02-14 13:22:56,758 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26577.21 MB 2025-02-14 13:22:56,758 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27114.08 MB 2025-02-14 13:22:56,758 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 536.87 MB 2025-02-14 13:22:56,758 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26216.34 MB 2025-02-14 13:22:56,776 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8157, cut from 8159 2025-02-14 13:22:56,777 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-14 13:22:56,783 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 13:22:56,783 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 13:22:56,783 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:22:56,783 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:22:56,783 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18630.00 MB 2025-02-14 13:22:56,783 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27064.62 MB 2025-02-14 13:22:56,783 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8434.62 MB 2025-02-14 13:22:56,783 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27114.08 MB 2025-02-14 13:22:56,783 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37595.64 MB 2025-02-14 13:22:56,783 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10481.57 MB 2025-02-14 13:22:56,783 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27064.62 MB 2025-02-14 13:22:56,952 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7949] 2025-02-14 13:22:56,954 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:22:56,954 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 13:22:56,955 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:22:56,955 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 13:22:56,960 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 13:22:56,961 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:22:56,961 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 13:22:56,961 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-14 13:23:20,570 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:23:20,570 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 13:23:20,578 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 13:23:20,584 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:23:20,584 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2781, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 13:23:20,586 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:23:20,586 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2781, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 13:24:03,768 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 13:24:03,768 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 13:24:03,768 - resource_logging.py:150 - __exit__ - DEBUG - Time: 43.17 seconds 2025-02-14 13:24:03,768 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:24:03,768 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32348.36 MB 2025-02-14 13:24:03,768 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42190.29 MB 2025-02-14 13:24:03,768 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 9841.93 MB 2025-02-14 13:24:03,768 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65361.94 MB 2025-02-14 13:24:03,768 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46133.15 MB 2025-02-14 13:24:03,768 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19228.79 MB 2025-02-14 13:24:03,768 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52032.09 MB 2025-02-14 13:24:03,984 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 13:24:03,984 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 13:24:03,984 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 13:24:03,984 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:24:03,984 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42190.29 MB 2025-02-14 13:24:03,984 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30236.60 MB 2025-02-14 13:24:03,984 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -11953.69 MB 2025-02-14 13:24:03,984 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46133.15 MB 2025-02-14 13:24:03,984 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 84737.52 MB 2025-02-14 13:24:03,984 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 38604.37 MB 2025-02-14 13:24:03,984 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 71610.08 MB 2025-02-14 13:24:05,951 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 13:24:05,951 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 13:24:05,951 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.96 seconds 2025-02-14 13:24:05,951 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:24:05,951 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30236.60 MB 2025-02-14 13:24:05,951 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30767.44 MB 2025-02-14 13:24:05,951 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 13:24:05,951 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 84737.52 MB 2025-02-14 13:24:05,951 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32784.78 MB 2025-02-14 13:24:05,951 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -51952.75 MB 2025-02-14 13:24:05,951 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34747.81 MB 2025-02-14 13:24:05,965 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 13:24:05,966 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 13:24:05,966 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:24:05,966 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:24:05,966 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30767.44 MB 2025-02-14 13:24:05,966 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32656.97 MB 2025-02-14 13:24:05,966 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 13:24:05,966 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32784.78 MB 2025-02-14 13:24:05,966 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36087.79 MB 2025-02-14 13:24:05,966 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 13:24:05,966 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34074.40 MB 2025-02-14 13:24:06,174 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 13:24:06,175 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 13:24:06,175 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 13:24:06,175 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:24:06,175 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32656.97 MB 2025-02-14 13:24:06,175 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34898.83 MB 2025-02-14 13:24:06,175 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 13:24:06,175 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36087.79 MB 2025-02-14 13:24:06,175 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42693.82 MB 2025-02-14 13:24:06,175 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 13:24:06,175 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40443.11 MB 2025-02-14 13:24:06,175 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 13:24:06,175 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 13:24:06,175 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 13:24:06,175 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:24:06,175 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30767.44 MB 2025-02-14 13:24:06,175 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34898.83 MB 2025-02-14 13:24:06,175 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 13:24:06,175 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32784.78 MB 2025-02-14 13:24:06,175 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42693.82 MB 2025-02-14 13:24:06,176 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9909.04 MB 2025-02-14 13:24:06,176 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40443.11 MB 2025-02-14 13:24:06,345 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 13:24:06,345 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 13:24:06,345 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 13:24:06,345 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:24:06,345 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36432.37 MB 2025-02-14 13:24:06,345 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37199.37 MB 2025-02-14 13:24:06,345 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 13:24:06,345 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42693.82 MB 2025-02-14 13:24:06,345 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43109.06 MB 2025-02-14 13:24:06,345 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 13:24:06,345 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37907.16 MB 2025-02-14 13:24:06,364 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 13:24:06,364 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 13:24:06,364 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:24:06,364 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:24:06,364 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37612.26 MB 2025-02-14 13:24:06,364 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37841.25 MB 2025-02-14 13:24:06,364 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.99 MB 2025-02-14 13:24:06,364 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43109.06 MB 2025-02-14 13:24:06,364 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43109.06 MB 2025-02-14 13:24:06,364 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:24:06,364 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38090.17 MB 2025-02-14 13:24:06,365 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 13:24:06,365 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 13:24:06,365 - resource_logging.py:150 - __exit__ - DEBUG - Time: 45.78 seconds 2025-02-14 13:24:06,365 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:24:06,365 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22658.53 MB 2025-02-14 13:24:06,365 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38042.15 MB 2025-02-14 13:24:06,365 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 15383.62 MB 2025-02-14 13:24:06,365 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55671.00 MB 2025-02-14 13:24:06,365 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43109.06 MB 2025-02-14 13:24:06,365 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12561.94 MB 2025-02-14 13:24:06,365 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38090.17 MB 2025-02-14 13:24:06,637 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 13:24:06,637 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 13:24:06,637 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 13:24:06,637 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:24:06,637 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38042.15 MB 2025-02-14 13:24:06,637 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27660.25 MB 2025-02-14 13:24:06,637 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10381.90 MB 2025-02-14 13:24:06,637 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43109.06 MB 2025-02-14 13:24:06,637 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43109.06 MB 2025-02-14 13:24:06,637 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:24:06,637 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40551.67 MB 2025-02-14 13:24:06,655 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8155, cut from 8157 2025-02-14 13:24:06,655 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 13:24:06,661 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 13:24:06,661 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 13:24:06,661 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:24:06,661 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:24:06,661 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27660.25 MB 2025-02-14 13:24:06,661 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36091.72 MB 2025-02-14 13:24:06,661 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8431.46 MB 2025-02-14 13:24:06,661 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43109.06 MB 2025-02-14 13:24:06,661 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47301.26 MB 2025-02-14 13:24:06,661 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4192.21 MB 2025-02-14 13:24:06,661 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36091.72 MB 2025-02-14 13:24:06,824 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7947] 2025-02-14 13:24:06,826 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:24:06,826 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 13:24:06,827 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:24:06,827 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 13:24:06,831 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 13:24:06,833 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:24:06,833 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 13:24:06,833 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 13:24:26,828 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:24:26,828 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 13:24:26,833 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 13:24:26,836 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:24:26,836 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 364, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 13:24:26,837 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:24:26,837 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 364, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 13:24:32,538 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 13:24:32,538 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 13:24:32,539 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.70 seconds 2025-02-14 13:24:32,539 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:24:32,539 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15505.12 MB 2025-02-14 13:24:32,539 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16793.29 MB 2025-02-14 13:24:32,539 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1288.18 MB 2025-02-14 13:24:32,539 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55685.68 MB 2025-02-14 13:24:32,539 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20088.62 MB 2025-02-14 13:24:32,539 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35597.06 MB 2025-02-14 13:24:32,539 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25656.77 MB 2025-02-14 13:24:32,562 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 13:24:32,562 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 13:24:32,562 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:24:32,563 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:24:32,563 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16793.29 MB 2025-02-14 13:24:32,563 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17192.61 MB 2025-02-14 13:24:32,563 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 399.32 MB 2025-02-14 13:24:32,563 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20088.62 MB 2025-02-14 13:24:32,563 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24171.77 MB 2025-02-14 13:24:32,563 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4083.15 MB 2025-02-14 13:24:32,563 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21499.05 MB 2025-02-14 13:24:34,173 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 13:24:34,173 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 13:24:34,173 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.61 seconds 2025-02-14 13:24:34,173 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:24:34,173 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17192.61 MB 2025-02-14 13:24:34,173 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17633.21 MB 2025-02-14 13:24:34,173 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 440.60 MB 2025-02-14 13:24:34,173 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24171.77 MB 2025-02-14 13:24:34,173 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21135.10 MB 2025-02-14 13:24:34,173 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3036.68 MB 2025-02-14 13:24:34,173 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21618.89 MB 2025-02-14 13:24:34,185 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 13:24:34,185 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 13:24:34,185 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:24:34,185 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:24:34,185 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17633.21 MB 2025-02-14 13:24:34,185 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19203.45 MB 2025-02-14 13:24:34,185 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1570.24 MB 2025-02-14 13:24:34,185 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21135.10 MB 2025-02-14 13:24:34,185 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22703.77 MB 2025-02-14 13:24:34,185 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1568.67 MB 2025-02-14 13:24:34,185 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20379.92 MB 2025-02-14 13:24:34,411 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 13:24:34,411 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 13:24:34,411 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 13:24:34,411 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:24:34,411 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19203.45 MB 2025-02-14 13:24:34,411 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21065.25 MB 2025-02-14 13:24:34,411 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1861.80 MB 2025-02-14 13:24:34,411 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22703.77 MB 2025-02-14 13:24:34,412 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27801.94 MB 2025-02-14 13:24:34,412 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5098.18 MB 2025-02-14 13:24:34,412 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25671.19 MB 2025-02-14 13:24:34,412 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 13:24:34,412 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 13:24:34,412 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.24 seconds 2025-02-14 13:24:34,412 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:24:34,412 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17633.21 MB 2025-02-14 13:24:34,412 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21065.25 MB 2025-02-14 13:24:34,412 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3432.04 MB 2025-02-14 13:24:34,412 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21135.10 MB 2025-02-14 13:24:34,412 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27801.94 MB 2025-02-14 13:24:34,412 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6666.85 MB 2025-02-14 13:24:34,412 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25671.19 MB 2025-02-14 13:24:34,559 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 13:24:34,559 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 13:24:34,559 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-14 13:24:34,559 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:24:34,559 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22338.61 MB 2025-02-14 13:24:34,559 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22975.23 MB 2025-02-14 13:24:34,559 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 636.61 MB 2025-02-14 13:24:34,559 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27801.94 MB 2025-02-14 13:24:34,559 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28143.78 MB 2025-02-14 13:24:34,559 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 341.84 MB 2025-02-14 13:24:34,559 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23562.69 MB 2025-02-14 13:24:34,575 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 13:24:34,575 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 13:24:34,575 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:24:34,575 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:24:34,575 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23317.93 MB 2025-02-14 13:24:34,575 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23531.55 MB 2025-02-14 13:24:34,575 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 213.62 MB 2025-02-14 13:24:34,575 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28143.78 MB 2025-02-14 13:24:34,576 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28143.78 MB 2025-02-14 13:24:34,576 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:24:34,576 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23671.22 MB 2025-02-14 13:24:34,577 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 13:24:34,577 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 13:24:34,577 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.74 seconds 2025-02-14 13:24:34,577 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:24:34,577 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14236.91 MB 2025-02-14 13:24:34,577 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23732.01 MB 2025-02-14 13:24:34,577 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 9495.09 MB 2025-02-14 13:24:34,577 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55685.68 MB 2025-02-14 13:24:34,577 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28143.78 MB 2025-02-14 13:24:34,577 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27541.90 MB 2025-02-14 13:24:34,577 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23732.01 MB 2025-02-14 13:24:34,847 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 13:24:34,847 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 13:24:34,847 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 13:24:34,847 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:24:34,847 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23732.01 MB 2025-02-14 13:24:34,847 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26736.82 MB 2025-02-14 13:24:34,847 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3004.82 MB 2025-02-14 13:24:34,847 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28143.78 MB 2025-02-14 13:24:34,847 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28143.78 MB 2025-02-14 13:24:34,847 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:24:34,847 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27037.27 MB 2025-02-14 13:24:34,866 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8137, cut from 8139 2025-02-14 13:24:34,866 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 13:24:34,872 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 13:24:34,872 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 13:24:34,872 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:24:34,872 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:24:34,872 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18911.13 MB 2025-02-14 13:24:34,872 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27324.65 MB 2025-02-14 13:24:34,872 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8413.52 MB 2025-02-14 13:24:34,872 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28143.78 MB 2025-02-14 13:24:34,872 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38598.08 MB 2025-02-14 13:24:34,872 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10454.30 MB 2025-02-14 13:24:34,872 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27324.65 MB 2025-02-14 13:24:35,035 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7929] 2025-02-14 13:24:35,036 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:24:35,036 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 13:24:35,037 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:24:35,037 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 13:24:35,042 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 13:24:35,043 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:24:35,043 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 13:24:35,043 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 13:24:49,199 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:24:49,199 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 13:24:49,204 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 13:24:49,210 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:24:49,210 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 379, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 13:24:49,212 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:24:49,212 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 379, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 13:24:55,129 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 13:24:55,129 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 13:24:55,129 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.91 seconds 2025-02-14 13:24:55,129 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:24:55,129 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15609.64 MB 2025-02-14 13:24:55,129 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16951.82 MB 2025-02-14 13:24:55,129 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1342.18 MB 2025-02-14 13:24:55,129 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46961.52 MB 2025-02-14 13:24:55,129 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19818.09 MB 2025-02-14 13:24:55,129 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27143.44 MB 2025-02-14 13:24:55,129 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25760.49 MB 2025-02-14 13:24:55,155 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 13:24:55,155 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 13:24:55,155 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:24:55,155 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:24:55,155 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16951.82 MB 2025-02-14 13:24:55,155 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17531.10 MB 2025-02-14 13:24:55,155 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 579.28 MB 2025-02-14 13:24:55,155 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19818.09 MB 2025-02-14 13:24:55,155 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23775.41 MB 2025-02-14 13:24:55,156 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3957.33 MB 2025-02-14 13:24:55,156 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22175.70 MB 2025-02-14 13:24:56,951 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 13:24:56,951 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 13:24:56,951 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.79 seconds 2025-02-14 13:24:56,951 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:24:56,951 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17531.10 MB 2025-02-14 13:24:56,951 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18020.80 MB 2025-02-14 13:24:56,951 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 489.70 MB 2025-02-14 13:24:56,951 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23775.41 MB 2025-02-14 13:24:56,951 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19782.43 MB 2025-02-14 13:24:56,951 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3992.98 MB 2025-02-14 13:24:56,951 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21958.41 MB 2025-02-14 13:24:56,964 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 13:24:56,964 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 13:24:56,964 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:24:56,964 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:24:56,964 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18020.80 MB 2025-02-14 13:24:56,964 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19763.79 MB 2025-02-14 13:24:56,964 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1743.00 MB 2025-02-14 13:24:56,964 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19782.43 MB 2025-02-14 13:24:56,964 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22835.89 MB 2025-02-14 13:24:56,964 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3053.45 MB 2025-02-14 13:24:56,964 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21072.29 MB 2025-02-14 13:24:57,162 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 13:24:57,162 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 13:24:57,162 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 13:24:57,162 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:24:57,162 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19763.79 MB 2025-02-14 13:24:57,162 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21831.91 MB 2025-02-14 13:24:57,162 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2068.12 MB 2025-02-14 13:24:57,162 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22835.89 MB 2025-02-14 13:24:57,162 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28942.79 MB 2025-02-14 13:24:57,162 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6106.91 MB 2025-02-14 13:24:57,162 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26949.26 MB 2025-02-14 13:24:57,163 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 13:24:57,163 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 13:24:57,163 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 13:24:57,163 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:24:57,163 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18020.80 MB 2025-02-14 13:24:57,163 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21831.91 MB 2025-02-14 13:24:57,163 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3811.11 MB 2025-02-14 13:24:57,163 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19782.43 MB 2025-02-14 13:24:57,163 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28942.79 MB 2025-02-14 13:24:57,163 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9160.36 MB 2025-02-14 13:24:57,163 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26949.26 MB 2025-02-14 13:24:57,319 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 13:24:57,319 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 13:24:57,319 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 13:24:57,319 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:24:57,319 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23247.52 MB 2025-02-14 13:24:57,319 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23956.91 MB 2025-02-14 13:24:57,319 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 709.39 MB 2025-02-14 13:24:57,319 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28942.79 MB 2025-02-14 13:24:57,319 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29326.57 MB 2025-02-14 13:24:57,319 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 383.78 MB 2025-02-14 13:24:57,319 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24609.85 MB 2025-02-14 13:24:57,336 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 13:24:57,336 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 13:24:57,336 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:24:57,336 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:24:57,336 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24337.81 MB 2025-02-14 13:24:57,337 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24565.97 MB 2025-02-14 13:24:57,337 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.17 MB 2025-02-14 13:24:57,337 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29326.57 MB 2025-02-14 13:24:57,337 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29326.57 MB 2025-02-14 13:24:57,337 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:24:57,337 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24722.39 MB 2025-02-14 13:24:57,338 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 13:24:57,338 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 13:24:57,338 - resource_logging.py:150 - __exit__ - DEBUG - Time: 8.12 seconds 2025-02-14 13:24:57,338 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:24:57,338 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14289.17 MB 2025-02-14 13:24:57,338 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24767.05 MB 2025-02-14 13:24:57,338 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 10477.87 MB 2025-02-14 13:24:57,338 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46961.52 MB 2025-02-14 13:24:57,338 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29326.57 MB 2025-02-14 13:24:57,338 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17634.95 MB 2025-02-14 13:24:57,338 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24767.05 MB 2025-02-14 13:24:57,608 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 13:24:57,608 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 13:24:57,608 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 13:24:57,608 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:24:57,608 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24767.05 MB 2025-02-14 13:24:57,608 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19149.75 MB 2025-02-14 13:24:57,608 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5617.29 MB 2025-02-14 13:24:57,608 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29326.57 MB 2025-02-14 13:24:57,608 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29326.57 MB 2025-02-14 13:24:57,608 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:24:57,608 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27781.05 MB 2025-02-14 13:24:57,626 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 13:24:57,627 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-14 13:24:57,633 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 13:24:57,633 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 13:24:57,633 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:24:57,633 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:24:57,633 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19149.75 MB 2025-02-14 13:24:57,633 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27588.78 MB 2025-02-14 13:24:57,633 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 13:24:57,633 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29326.57 MB 2025-02-14 13:24:57,633 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39816.53 MB 2025-02-14 13:24:57,633 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 13:24:57,633 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27588.78 MB 2025-02-14 13:24:57,795 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 13:24:57,796 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:24:57,796 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 13:24:57,797 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:24:57,797 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 13:24:57,802 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 13:24:57,803 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:24:57,803 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 13:24:57,803 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-14 13:25:57,083 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:25:57,083 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 13:25:57,088 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 13:25:57,092 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:25:57,092 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 234, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 13:25:57,094 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:25:57,094 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 234, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 13:26:00,711 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 13:26:00,711 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 13:26:00,711 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.61 seconds 2025-02-14 13:26:00,711 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:26:00,711 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14599.90 MB 2025-02-14 13:26:00,711 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15428.02 MB 2025-02-14 13:26:00,711 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 828.11 MB 2025-02-14 13:26:00,711 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52401.54 MB 2025-02-14 13:26:00,711 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18217.96 MB 2025-02-14 13:26:00,711 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34183.58 MB 2025-02-14 13:26:00,711 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24298.57 MB 2025-02-14 13:26:00,729 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 13:26:00,729 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 13:26:00,729 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:26:00,729 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:26:00,729 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15428.02 MB 2025-02-14 13:26:00,729 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15745.68 MB 2025-02-14 13:26:00,729 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 317.66 MB 2025-02-14 13:26:00,729 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18217.96 MB 2025-02-14 13:26:00,729 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19790.82 MB 2025-02-14 13:26:00,729 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1572.86 MB 2025-02-14 13:26:00,729 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18583.20 MB 2025-02-14 13:26:01,795 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 13:26:01,795 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 13:26:01,795 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.06 seconds 2025-02-14 13:26:01,795 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:26:01,795 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15745.68 MB 2025-02-14 13:26:01,795 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16040.30 MB 2025-02-14 13:26:01,795 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 294.62 MB 2025-02-14 13:26:01,795 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19790.82 MB 2025-02-14 13:26:01,795 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18970.84 MB 2025-02-14 13:26:01,795 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -819.99 MB 2025-02-14 13:26:01,795 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20002.09 MB 2025-02-14 13:26:01,803 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 13:26:01,804 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 13:26:01,804 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:26:01,804 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:26:01,804 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16040.30 MB 2025-02-14 13:26:01,804 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17088.73 MB 2025-02-14 13:26:01,804 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1048.44 MB 2025-02-14 13:26:01,804 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18970.84 MB 2025-02-14 13:26:01,804 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18970.84 MB 2025-02-14 13:26:01,804 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:26:01,804 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17875.41 MB 2025-02-14 13:26:01,924 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 13:26:01,924 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 13:26:01,924 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 13:26:01,924 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:26:01,924 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17088.73 MB 2025-02-14 13:26:01,924 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18333.78 MB 2025-02-14 13:26:01,924 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1245.05 MB 2025-02-14 13:26:01,924 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18970.84 MB 2025-02-14 13:26:01,924 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22640.85 MB 2025-02-14 13:26:01,924 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3670.02 MB 2025-02-14 13:26:01,924 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21413.19 MB 2025-02-14 13:26:01,925 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 13:26:01,925 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 13:26:01,925 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 13:26:01,925 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:26:01,925 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16040.30 MB 2025-02-14 13:26:01,925 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18333.78 MB 2025-02-14 13:26:01,925 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2293.48 MB 2025-02-14 13:26:01,925 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18970.84 MB 2025-02-14 13:26:01,925 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22640.85 MB 2025-02-14 13:26:01,925 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3670.02 MB 2025-02-14 13:26:01,925 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21413.19 MB 2025-02-14 13:26:02,017 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 13:26:02,017 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 13:26:02,017 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 13:26:02,017 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:26:02,017 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19184.90 MB 2025-02-14 13:26:02,017 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19610.84 MB 2025-02-14 13:26:02,017 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 425.95 MB 2025-02-14 13:26:02,017 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22640.85 MB 2025-02-14 13:26:02,017 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22869.44 MB 2025-02-14 13:26:02,017 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 228.59 MB 2025-02-14 13:26:02,017 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20004.79 MB 2025-02-14 13:26:02,029 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 13:26:02,029 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 13:26:02,029 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:26:02,029 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:26:02,029 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19840.00 MB 2025-02-14 13:26:02,029 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20060.91 MB 2025-02-14 13:26:02,029 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 220.91 MB 2025-02-14 13:26:02,029 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22869.44 MB 2025-02-14 13:26:02,029 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22869.44 MB 2025-02-14 13:26:02,029 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:26:02,029 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20135.04 MB 2025-02-14 13:26:02,030 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 13:26:02,030 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 13:26:02,030 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.93 seconds 2025-02-14 13:26:02,030 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:26:02,030 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13784.63 MB 2025-02-14 13:26:02,030 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20261.37 MB 2025-02-14 13:26:02,030 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6476.74 MB 2025-02-14 13:26:02,030 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52401.54 MB 2025-02-14 13:26:02,030 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22869.44 MB 2025-02-14 13:26:02,030 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29532.09 MB 2025-02-14 13:26:02,030 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20261.37 MB 2025-02-14 13:26:02,297 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 13:26:02,297 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 13:26:02,297 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 13:26:02,297 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:26:02,297 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14934.91 MB 2025-02-14 13:26:02,297 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17939.72 MB 2025-02-14 13:26:02,297 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3004.82 MB 2025-02-14 13:26:02,297 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22869.44 MB 2025-02-14 13:26:02,297 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22869.44 MB 2025-02-14 13:26:02,297 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:26:02,297 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18240.17 MB 2025-02-14 13:26:02,315 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8137, cut from 8139 2025-02-14 13:26:02,316 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 13:26:02,322 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 13:26:02,322 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 13:26:02,322 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:26:02,322 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:26:02,322 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17939.72 MB 2025-02-14 13:26:02,322 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26353.25 MB 2025-02-14 13:26:02,322 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8413.52 MB 2025-02-14 13:26:02,322 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22869.44 MB 2025-02-14 13:26:02,322 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33323.75 MB 2025-02-14 13:26:02,322 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10454.30 MB 2025-02-14 13:26:02,322 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26353.25 MB 2025-02-14 13:26:02,474 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7929] 2025-02-14 13:26:02,476 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:26:02,476 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 13:26:02,477 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:26:02,477 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 13:26:02,481 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 13:26:02,482 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:26:02,482 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 13:26:02,482 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 13:27:17,042 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:27:17,042 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 13:27:17,047 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 13:27:17,052 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:27:17,052 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1308, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 13:27:17,054 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:27:17,054 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1308, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 13:27:37,117 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 13:27:37,118 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 13:27:37,118 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.05 seconds 2025-02-14 13:27:37,118 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:27:37,118 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22083.06 MB 2025-02-14 13:27:37,118 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26712.00 MB 2025-02-14 13:27:37,118 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4628.94 MB 2025-02-14 13:27:37,118 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41687.19 MB 2025-02-14 13:27:37,118 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38266.73 MB 2025-02-14 13:27:37,118 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3420.45 MB 2025-02-14 13:27:37,118 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35631.29 MB 2025-02-14 13:27:37,195 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 13:27:37,195 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 13:27:37,195 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 13:27:37,195 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:27:37,195 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26712.00 MB 2025-02-14 13:27:37,195 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22577.73 MB 2025-02-14 13:27:37,195 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4134.26 MB 2025-02-14 13:27:37,195 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38266.73 MB 2025-02-14 13:27:37,195 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46854.57 MB 2025-02-14 13:27:37,195 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8587.84 MB 2025-02-14 13:27:37,195 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39661.23 MB 2025-02-14 13:27:39,101 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 13:27:39,101 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 13:27:39,101 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.90 seconds 2025-02-14 13:27:39,101 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:27:39,101 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22577.73 MB 2025-02-14 13:27:39,101 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23108.58 MB 2025-02-14 13:27:39,101 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 13:27:39,101 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46854.57 MB 2025-02-14 13:27:39,101 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33636.22 MB 2025-02-14 13:27:39,101 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13218.35 MB 2025-02-14 13:27:39,101 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27087.91 MB 2025-02-14 13:27:39,114 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 13:27:39,114 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 13:27:39,114 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:27:39,114 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:27:39,114 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23108.58 MB 2025-02-14 13:27:39,114 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24998.11 MB 2025-02-14 13:27:39,114 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 13:27:39,114 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33636.22 MB 2025-02-14 13:27:39,114 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33636.22 MB 2025-02-14 13:27:39,114 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:27:39,114 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26415.54 MB 2025-02-14 13:27:39,328 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 13:27:39,328 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 13:27:39,328 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 13:27:39,328 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:27:39,328 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24998.11 MB 2025-02-14 13:27:39,328 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27239.97 MB 2025-02-14 13:27:39,328 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 13:27:39,328 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33636.22 MB 2025-02-14 13:27:39,328 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35995.52 MB 2025-02-14 13:27:39,328 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2359.30 MB 2025-02-14 13:27:39,328 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32784.25 MB 2025-02-14 13:27:39,328 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 13:27:39,329 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 13:27:39,329 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 13:27:39,329 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:27:39,329 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23108.58 MB 2025-02-14 13:27:39,329 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27239.97 MB 2025-02-14 13:27:39,329 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 13:27:39,329 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33636.22 MB 2025-02-14 13:27:39,329 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35995.52 MB 2025-02-14 13:27:39,329 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2359.30 MB 2025-02-14 13:27:39,329 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32784.25 MB 2025-02-14 13:27:39,503 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 13:27:39,503 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 13:27:39,503 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 13:27:39,503 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:27:39,503 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28773.51 MB 2025-02-14 13:27:39,503 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29540.51 MB 2025-02-14 13:27:39,503 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 13:27:39,503 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35995.52 MB 2025-02-14 13:27:39,503 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36410.75 MB 2025-02-14 13:27:39,503 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 13:27:39,504 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30248.30 MB 2025-02-14 13:27:39,523 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 13:27:39,523 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 13:27:39,523 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:27:39,523 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:27:39,523 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29953.40 MB 2025-02-14 13:27:39,523 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30182.82 MB 2025-02-14 13:27:39,523 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.42 MB 2025-02-14 13:27:39,523 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36410.75 MB 2025-02-14 13:27:39,523 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36410.75 MB 2025-02-14 13:27:39,523 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:27:39,523 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30390.97 MB 2025-02-14 13:27:39,524 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 13:27:39,524 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 13:27:39,524 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.47 seconds 2025-02-14 13:27:39,524 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:27:39,524 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17525.88 MB 2025-02-14 13:27:39,524 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30383.84 MB 2025-02-14 13:27:39,524 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12857.96 MB 2025-02-14 13:27:39,524 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41687.19 MB 2025-02-14 13:27:39,524 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36410.75 MB 2025-02-14 13:27:39,524 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5276.43 MB 2025-02-14 13:27:39,524 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30390.97 MB 2025-02-14 13:27:39,794 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 13:27:39,795 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 13:27:39,795 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 13:27:39,795 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:27:39,795 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30383.84 MB 2025-02-14 13:27:39,795 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22529.61 MB 2025-02-14 13:27:39,795 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7854.23 MB 2025-02-14 13:27:39,795 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36410.75 MB 2025-02-14 13:27:39,795 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36410.75 MB 2025-02-14 13:27:39,795 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:27:39,795 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32894.90 MB 2025-02-14 13:27:39,813 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8160, cut from 8162 2025-02-14 13:27:39,813 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 13:27:39,819 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 13:27:39,819 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 13:27:39,819 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:27:39,819 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:27:39,819 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22529.61 MB 2025-02-14 13:27:39,819 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30967.08 MB 2025-02-14 13:27:39,819 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8437.47 MB 2025-02-14 13:27:39,819 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36410.75 MB 2025-02-14 13:27:39,819 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44799.36 MB 2025-02-14 13:27:39,819 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8388.61 MB 2025-02-14 13:27:39,819 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30967.08 MB 2025-02-14 13:27:39,983 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7952] 2025-02-14 13:27:39,984 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:27:39,984 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 13:27:39,985 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:27:39,985 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 13:27:39,990 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 13:27:39,991 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:27:39,991 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 13:27:39,991 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 13:28:26,268 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:28:26,269 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 13:28:26,273 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 13:28:26,277 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:28:26,277 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1987, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 13:28:26,278 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:28:26,278 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1987, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 13:28:57,100 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 13:28:57,100 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 13:28:57,100 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.81 seconds 2025-02-14 13:28:57,100 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:28:57,100 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26814.44 MB 2025-02-14 13:28:57,100 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33846.32 MB 2025-02-14 13:28:57,100 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7031.88 MB 2025-02-14 13:28:57,100 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53187.97 MB 2025-02-14 13:28:57,100 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40672.17 MB 2025-02-14 13:28:57,100 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12515.80 MB 2025-02-14 13:28:57,100 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42854.90 MB 2025-02-14 13:28:57,229 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 13:28:57,229 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 13:28:57,229 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 13:28:57,229 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:28:57,229 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33846.32 MB 2025-02-14 13:28:57,229 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26107.64 MB 2025-02-14 13:28:57,229 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7738.68 MB 2025-02-14 13:28:57,229 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40672.17 MB 2025-02-14 13:28:57,229 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 62759.37 MB 2025-02-14 13:28:57,229 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 22087.20 MB 2025-02-14 13:28:57,229 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 53212.54 MB 2025-02-14 13:28:59,187 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 13:28:59,187 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 13:28:59,187 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.96 seconds 2025-02-14 13:28:59,187 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:28:59,187 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26107.64 MB 2025-02-14 13:28:59,187 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26638.49 MB 2025-02-14 13:28:59,187 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 13:28:59,187 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62759.37 MB 2025-02-14 13:28:59,187 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30872.17 MB 2025-02-14 13:28:59,187 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31887.20 MB 2025-02-14 13:28:59,187 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30618.86 MB 2025-02-14 13:28:59,201 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 13:28:59,201 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 13:28:59,201 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:28:59,201 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:28:59,201 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26638.49 MB 2025-02-14 13:28:59,201 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28528.02 MB 2025-02-14 13:28:59,201 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 13:28:59,201 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30872.17 MB 2025-02-14 13:28:59,201 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32759.61 MB 2025-02-14 13:28:59,201 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 13:28:59,201 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29945.45 MB 2025-02-14 13:28:59,421 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 13:28:59,421 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 13:28:59,421 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 13:28:59,421 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:28:59,421 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28528.02 MB 2025-02-14 13:28:59,421 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30769.88 MB 2025-02-14 13:28:59,421 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 13:28:59,421 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32759.61 MB 2025-02-14 13:28:59,421 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38893.78 MB 2025-02-14 13:28:59,421 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 13:28:59,421 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36314.16 MB 2025-02-14 13:28:59,421 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 13:28:59,421 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 13:28:59,421 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 13:28:59,421 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:28:59,421 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26638.49 MB 2025-02-14 13:28:59,421 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30769.88 MB 2025-02-14 13:28:59,422 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 13:28:59,422 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30872.17 MB 2025-02-14 13:28:59,422 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38893.78 MB 2025-02-14 13:28:59,422 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-14 13:28:59,422 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36314.16 MB 2025-02-14 13:28:59,604 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 13:28:59,604 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 13:28:59,604 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-14 13:28:59,604 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:28:59,604 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32303.42 MB 2025-02-14 13:28:59,604 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33070.42 MB 2025-02-14 13:28:59,604 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 13:28:59,604 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38893.78 MB 2025-02-14 13:28:59,604 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39306.92 MB 2025-02-14 13:28:59,604 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 13:28:59,604 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33778.21 MB 2025-02-14 13:28:59,623 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 13:28:59,624 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 13:28:59,624 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:28:59,624 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:28:59,624 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33483.31 MB 2025-02-14 13:28:59,624 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33711.29 MB 2025-02-14 13:28:59,624 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.98 MB 2025-02-14 13:28:59,624 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39306.92 MB 2025-02-14 13:28:59,624 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39306.92 MB 2025-02-14 13:28:59,624 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:28:59,624 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33913.45 MB 2025-02-14 13:28:59,625 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 13:28:59,625 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 13:28:59,625 - resource_logging.py:150 - __exit__ - DEBUG - Time: 33.34 seconds 2025-02-14 13:28:59,625 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:28:59,625 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19891.57 MB 2025-02-14 13:28:59,625 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33911.18 MB 2025-02-14 13:28:59,625 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14019.60 MB 2025-02-14 13:28:59,625 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53187.97 MB 2025-02-14 13:28:59,625 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39306.92 MB 2025-02-14 13:28:59,625 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13881.05 MB 2025-02-14 13:28:59,625 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33913.45 MB 2025-02-14 13:28:59,895 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 13:28:59,895 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 13:28:59,895 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 13:28:59,895 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:28:59,895 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33911.18 MB 2025-02-14 13:28:59,895 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24877.68 MB 2025-02-14 13:28:59,895 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9033.50 MB 2025-02-14 13:28:59,895 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39306.92 MB 2025-02-14 13:28:59,895 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39306.92 MB 2025-02-14 13:28:59,895 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:28:59,895 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36408.10 MB 2025-02-14 13:28:59,913 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8114, cut from 8116 2025-02-14 13:28:59,913 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 13:28:59,920 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 13:28:59,920 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 13:28:59,920 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:28:59,920 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:28:59,920 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24877.68 MB 2025-02-14 13:28:59,920 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33266.82 MB 2025-02-14 13:28:59,920 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8389.15 MB 2025-02-14 13:28:59,920 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39306.92 MB 2025-02-14 13:28:59,920 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47649.39 MB 2025-02-14 13:28:59,920 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8342.47 MB 2025-02-14 13:28:59,920 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33266.82 MB 2025-02-14 13:29:00,089 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7906] 2025-02-14 13:29:00,090 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:29:00,090 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 13:29:00,091 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:29:00,091 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 13:29:00,096 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 13:29:00,097 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:29:00,097 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 13:29:00,097 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 13:29:58,788 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:29:58,788 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 13:29:58,793 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 13:29:58,797 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:29:58,797 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1193, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 13:29:58,798 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:29:58,798 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1193, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 13:30:17,369 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 13:30:17,370 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 13:30:17,370 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.56 seconds 2025-02-14 13:30:17,370 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:30:17,370 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21281.72 MB 2025-02-14 13:30:17,370 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25503.68 MB 2025-02-14 13:30:17,370 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4221.96 MB 2025-02-14 13:30:17,370 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55991.86 MB 2025-02-14 13:30:17,370 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29483.86 MB 2025-02-14 13:30:17,370 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26508.00 MB 2025-02-14 13:30:17,370 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34377.78 MB 2025-02-14 13:30:17,455 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 13:30:17,455 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 13:30:17,455 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 13:30:17,455 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:30:17,455 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25503.68 MB 2025-02-14 13:30:17,455 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21979.89 MB 2025-02-14 13:30:17,455 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3523.80 MB 2025-02-14 13:30:17,455 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29483.86 MB 2025-02-14 13:30:17,455 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45459.96 MB 2025-02-14 13:30:17,455 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 15976.10 MB 2025-02-14 13:30:17,455 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38139.64 MB 2025-02-14 13:30:19,385 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 13:30:19,385 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 13:30:19,385 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 13:30:19,385 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:30:19,385 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21979.89 MB 2025-02-14 13:30:19,385 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22510.73 MB 2025-02-14 13:30:19,385 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 13:30:19,385 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45459.96 MB 2025-02-14 13:30:19,385 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26675.77 MB 2025-02-14 13:30:19,385 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18784.19 MB 2025-02-14 13:30:19,385 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26490.06 MB 2025-02-14 13:30:19,399 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 13:30:19,399 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 13:30:19,399 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:30:19,399 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:30:19,399 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22510.73 MB 2025-02-14 13:30:19,399 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24400.26 MB 2025-02-14 13:30:19,400 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 13:30:19,400 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26675.77 MB 2025-02-14 13:30:19,400 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28563.21 MB 2025-02-14 13:30:19,400 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 13:30:19,400 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25817.69 MB 2025-02-14 13:30:19,617 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 13:30:19,617 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 13:30:19,617 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 13:30:19,617 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:30:19,617 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24400.26 MB 2025-02-14 13:30:19,617 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26642.12 MB 2025-02-14 13:30:19,617 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 13:30:19,617 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28563.21 MB 2025-02-14 13:30:19,617 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34225.52 MB 2025-02-14 13:30:19,617 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 13:30:19,617 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32186.40 MB 2025-02-14 13:30:19,618 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 13:30:19,618 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 13:30:19,618 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 13:30:19,618 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:30:19,618 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22510.73 MB 2025-02-14 13:30:19,618 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26642.12 MB 2025-02-14 13:30:19,618 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 13:30:19,618 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26675.77 MB 2025-02-14 13:30:19,618 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34225.52 MB 2025-02-14 13:30:19,618 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-14 13:30:19,618 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32186.40 MB 2025-02-14 13:30:19,802 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 13:30:19,802 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 13:30:19,802 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-14 13:30:19,802 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:30:19,802 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28175.66 MB 2025-02-14 13:30:19,802 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28942.66 MB 2025-02-14 13:30:19,802 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 13:30:19,802 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34225.52 MB 2025-02-14 13:30:19,802 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34642.85 MB 2025-02-14 13:30:19,802 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 13:30:19,802 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29650.45 MB 2025-02-14 13:30:19,821 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 13:30:19,822 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 13:30:19,822 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:30:19,822 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:30:19,822 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29355.55 MB 2025-02-14 13:30:19,822 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29584.28 MB 2025-02-14 13:30:19,822 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.73 MB 2025-02-14 13:30:19,822 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34642.85 MB 2025-02-14 13:30:19,822 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34642.85 MB 2025-02-14 13:30:19,822 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:30:19,822 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29828.56 MB 2025-02-14 13:30:19,823 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 13:30:19,823 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 13:30:19,823 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.02 seconds 2025-02-14 13:30:19,823 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:30:19,823 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17125.21 MB 2025-02-14 13:30:19,823 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29784.83 MB 2025-02-14 13:30:19,823 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12659.62 MB 2025-02-14 13:30:19,823 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55991.86 MB 2025-02-14 13:30:19,823 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34642.85 MB 2025-02-14 13:30:19,823 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21349.01 MB 2025-02-14 13:30:19,823 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29828.56 MB 2025-02-14 13:30:20,092 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 13:30:20,093 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 13:30:20,093 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 13:30:20,093 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:30:20,093 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29784.83 MB 2025-02-14 13:30:20,093 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22121.60 MB 2025-02-14 13:30:20,093 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7663.23 MB 2025-02-14 13:30:20,093 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34642.85 MB 2025-02-14 13:30:20,093 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34642.85 MB 2025-02-14 13:30:20,093 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:30:20,093 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32290.05 MB 2025-02-14 13:30:20,111 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8141, cut from 8143 2025-02-14 13:30:20,111 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 13:30:20,117 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 13:30:20,117 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 13:30:20,117 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:30:20,117 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:30:20,117 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22121.60 MB 2025-02-14 13:30:20,117 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30539.35 MB 2025-02-14 13:30:20,117 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8417.74 MB 2025-02-14 13:30:20,117 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34642.85 MB 2025-02-14 13:30:20,117 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43010.49 MB 2025-02-14 13:30:20,117 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8367.64 MB 2025-02-14 13:30:20,117 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30539.35 MB 2025-02-14 13:30:20,281 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7933] 2025-02-14 13:30:20,282 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:30:20,282 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 13:30:20,283 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:30:20,283 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 13:30:20,288 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 13:30:20,289 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:30:20,289 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 13:30:20,289 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 13:31:20,843 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:31:20,844 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 13:31:20,852 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 13:31:20,861 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:31:20,861 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1552, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 13:31:20,863 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:31:20,863 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1552, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 13:31:45,087 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 13:31:45,087 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 13:31:45,087 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.21 seconds 2025-02-14 13:31:45,087 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:31:45,087 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23783.29 MB 2025-02-14 13:31:45,087 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29275.73 MB 2025-02-14 13:31:45,087 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5492.44 MB 2025-02-14 13:31:45,087 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51378.13 MB 2025-02-14 13:31:45,087 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39122.37 MB 2025-02-14 13:31:45,087 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12255.76 MB 2025-02-14 13:31:45,087 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38238.30 MB 2025-02-14 13:31:45,177 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 13:31:45,177 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 13:31:45,177 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 13:31:45,177 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:31:45,177 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29275.73 MB 2025-02-14 13:31:45,177 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23846.21 MB 2025-02-14 13:31:45,177 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5429.52 MB 2025-02-14 13:31:45,177 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39122.37 MB 2025-02-14 13:31:45,177 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49239.03 MB 2025-02-14 13:31:45,177 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10116.66 MB 2025-02-14 13:31:45,177 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44193.64 MB 2025-02-14 13:31:47,107 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 13:31:47,107 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 13:31:47,107 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 13:31:47,107 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:31:47,107 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23846.21 MB 2025-02-14 13:31:47,107 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24377.06 MB 2025-02-14 13:31:47,107 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 13:31:47,107 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49239.03 MB 2025-02-14 13:31:47,107 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29458.69 MB 2025-02-14 13:31:47,107 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19780.34 MB 2025-02-14 13:31:47,107 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28356.39 MB 2025-02-14 13:31:47,122 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 13:31:47,122 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 13:31:47,122 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:31:47,122 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:31:47,122 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24377.06 MB 2025-02-14 13:31:47,122 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26266.59 MB 2025-02-14 13:31:47,122 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 13:31:47,122 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29458.69 MB 2025-02-14 13:31:47,122 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30402.41 MB 2025-02-14 13:31:47,122 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 13:31:47,122 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27684.02 MB 2025-02-14 13:31:47,334 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 13:31:47,334 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 13:31:47,334 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 13:31:47,334 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:31:47,334 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26266.59 MB 2025-02-14 13:31:47,334 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28508.45 MB 2025-02-14 13:31:47,334 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 13:31:47,334 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30402.41 MB 2025-02-14 13:31:47,334 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36064.72 MB 2025-02-14 13:31:47,334 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 13:31:47,334 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34052.73 MB 2025-02-14 13:31:47,335 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 13:31:47,335 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 13:31:47,335 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 13:31:47,335 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:31:47,335 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24377.06 MB 2025-02-14 13:31:47,335 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28508.45 MB 2025-02-14 13:31:47,335 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 13:31:47,335 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29458.69 MB 2025-02-14 13:31:47,335 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36064.72 MB 2025-02-14 13:31:47,335 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 13:31:47,335 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34052.73 MB 2025-02-14 13:31:47,505 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 13:31:47,505 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 13:31:47,505 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 13:31:47,505 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:31:47,505 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30041.99 MB 2025-02-14 13:31:47,505 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30808.99 MB 2025-02-14 13:31:47,505 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 13:31:47,505 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36064.72 MB 2025-02-14 13:31:47,505 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36479.96 MB 2025-02-14 13:31:47,505 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 13:31:47,505 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31516.78 MB 2025-02-14 13:31:47,524 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 13:31:47,524 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 13:31:47,524 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:31:47,524 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:31:47,524 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31221.88 MB 2025-02-14 13:31:47,524 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31452.56 MB 2025-02-14 13:31:47,524 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 230.69 MB 2025-02-14 13:31:47,524 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36479.96 MB 2025-02-14 13:31:47,524 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36479.96 MB 2025-02-14 13:31:47,524 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:31:47,524 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31654.31 MB 2025-02-14 13:31:47,525 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 13:31:47,525 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 13:31:47,525 - resource_logging.py:150 - __exit__ - DEBUG - Time: 26.66 seconds 2025-02-14 13:31:47,525 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:31:47,525 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18376.00 MB 2025-02-14 13:31:47,525 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31653.64 MB 2025-02-14 13:31:47,525 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13277.64 MB 2025-02-14 13:31:47,525 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51378.13 MB 2025-02-14 13:31:47,525 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36479.96 MB 2025-02-14 13:31:47,525 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14898.17 MB 2025-02-14 13:31:47,525 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31654.31 MB 2025-02-14 13:31:47,795 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 13:31:47,795 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 13:31:47,795 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 13:31:47,795 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:31:47,795 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31653.64 MB 2025-02-14 13:31:47,795 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23380.39 MB 2025-02-14 13:31:47,795 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8273.25 MB 2025-02-14 13:31:47,795 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36479.96 MB 2025-02-14 13:31:47,795 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36479.96 MB 2025-02-14 13:31:47,795 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:31:47,795 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34165.30 MB 2025-02-14 13:31:47,813 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 13:31:47,813 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 13:31:47,819 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 13:31:47,819 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 13:31:47,819 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:31:47,819 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:31:47,820 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23380.39 MB 2025-02-14 13:31:47,820 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31819.41 MB 2025-02-14 13:31:47,820 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 13:31:47,820 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36479.96 MB 2025-02-14 13:31:47,820 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44870.66 MB 2025-02-14 13:31:47,820 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 13:31:47,820 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31819.41 MB 2025-02-14 13:31:47,983 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 13:31:47,984 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:31:47,984 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 13:31:47,985 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:31:47,985 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 13:31:47,990 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 13:31:47,991 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:31:47,991 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 13:31:47,991 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 13:32:26,986 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:32:26,986 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 13:32:26,991 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 13:32:26,995 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:32:26,995 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1370, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 13:32:26,996 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:32:26,996 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1370, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 13:32:48,217 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 13:32:48,217 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 13:32:48,218 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.21 seconds 2025-02-14 13:32:48,218 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:32:48,218 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22515.09 MB 2025-02-14 13:32:48,218 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27363.70 MB 2025-02-14 13:32:48,218 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4848.62 MB 2025-02-14 13:32:48,218 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57455.67 MB 2025-02-14 13:32:48,218 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38503.71 MB 2025-02-14 13:32:48,218 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18951.96 MB 2025-02-14 13:32:48,218 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36289.81 MB 2025-02-14 13:32:48,302 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 13:32:48,302 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 13:32:48,302 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 13:32:48,302 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:32:48,302 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27363.70 MB 2025-02-14 13:32:48,302 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22900.05 MB 2025-02-14 13:32:48,302 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4463.65 MB 2025-02-14 13:32:48,302 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38503.71 MB 2025-02-14 13:32:48,302 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48035.27 MB 2025-02-14 13:32:48,303 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9531.56 MB 2025-02-14 13:32:48,303 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41732.47 MB 2025-02-14 13:32:50,232 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 13:32:50,232 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 13:32:50,232 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 13:32:50,232 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:32:50,232 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22900.05 MB 2025-02-14 13:32:50,232 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23430.89 MB 2025-02-14 13:32:50,232 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 13:32:50,232 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48035.27 MB 2025-02-14 13:32:50,232 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33655.10 MB 2025-02-14 13:32:50,232 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14380.17 MB 2025-02-14 13:32:50,232 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27410.23 MB 2025-02-14 13:32:50,245 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 13:32:50,245 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 13:32:50,245 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:32:50,245 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:32:50,245 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23430.89 MB 2025-02-14 13:32:50,245 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25320.43 MB 2025-02-14 13:32:50,245 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 13:32:50,245 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33655.10 MB 2025-02-14 13:32:50,245 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33655.10 MB 2025-02-14 13:32:50,245 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:32:50,245 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26737.86 MB 2025-02-14 13:32:50,458 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 13:32:50,458 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 13:32:50,458 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 13:32:50,458 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:32:50,458 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25320.43 MB 2025-02-14 13:32:50,458 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27562.28 MB 2025-02-14 13:32:50,458 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 13:32:50,458 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33655.10 MB 2025-02-14 13:32:50,458 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37429.97 MB 2025-02-14 13:32:50,458 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 13:32:50,458 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33106.57 MB 2025-02-14 13:32:50,459 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 13:32:50,459 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 13:32:50,459 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 13:32:50,459 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:32:50,459 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23430.89 MB 2025-02-14 13:32:50,459 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27562.28 MB 2025-02-14 13:32:50,459 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 13:32:50,459 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33655.10 MB 2025-02-14 13:32:50,459 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37429.97 MB 2025-02-14 13:32:50,459 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 13:32:50,459 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33106.57 MB 2025-02-14 13:32:50,631 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 13:32:50,631 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 13:32:50,631 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 13:32:50,631 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:32:50,631 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29095.83 MB 2025-02-14 13:32:50,631 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29862.83 MB 2025-02-14 13:32:50,631 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 13:32:50,631 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37429.97 MB 2025-02-14 13:32:50,631 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37843.11 MB 2025-02-14 13:32:50,631 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 13:32:50,631 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30570.62 MB 2025-02-14 13:32:50,650 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 13:32:50,650 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 13:32:50,650 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:32:50,650 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:32:50,650 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30275.72 MB 2025-02-14 13:32:50,650 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30504.78 MB 2025-02-14 13:32:50,650 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.06 MB 2025-02-14 13:32:50,650 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37843.11 MB 2025-02-14 13:32:50,650 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37843.11 MB 2025-02-14 13:32:50,650 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:32:50,650 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30749.48 MB 2025-02-14 13:32:50,651 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 13:32:50,651 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 13:32:50,651 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.65 seconds 2025-02-14 13:32:50,651 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:32:50,651 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17741.90 MB 2025-02-14 13:32:50,651 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30705.75 MB 2025-02-14 13:32:50,651 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12963.86 MB 2025-02-14 13:32:50,651 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57455.67 MB 2025-02-14 13:32:50,651 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37843.11 MB 2025-02-14 13:32:50,651 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19612.57 MB 2025-02-14 13:32:50,651 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30749.48 MB 2025-02-14 13:32:50,922 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 13:32:50,922 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 13:32:50,922 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 13:32:50,922 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:32:50,922 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30705.75 MB 2025-02-14 13:32:50,922 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22744.76 MB 2025-02-14 13:32:50,922 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7960.99 MB 2025-02-14 13:32:50,922 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37843.11 MB 2025-02-14 13:32:50,922 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37843.11 MB 2025-02-14 13:32:50,922 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:32:50,922 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33216.19 MB 2025-02-14 13:32:50,940 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8158, cut from 8160 2025-02-14 13:32:50,940 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 13:32:50,946 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 13:32:50,946 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 13:32:50,946 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:32:50,946 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:32:50,946 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22744.76 MB 2025-02-14 13:32:50,946 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31179.61 MB 2025-02-14 13:32:50,947 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8434.85 MB 2025-02-14 13:32:50,947 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37843.11 MB 2025-02-14 13:32:50,947 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46229.62 MB 2025-02-14 13:32:50,947 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8386.51 MB 2025-02-14 13:32:50,947 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31179.61 MB 2025-02-14 13:32:51,109 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7950] 2025-02-14 13:32:51,110 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:32:51,110 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 13:32:51,111 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:32:51,111 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 13:32:51,116 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 13:32:51,117 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:32:51,117 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 13:32:51,117 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 13:33:05,386 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:33:05,387 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 13:33:05,392 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 13:33:05,395 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:33:05,395 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 875, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 13:33:05,396 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:33:05,396 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 875, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 13:33:19,111 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 13:33:19,112 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 13:33:19,112 - resource_logging.py:150 - __exit__ - DEBUG - Time: 13.71 seconds 2025-02-14 13:33:19,112 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:33:19,112 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19065.85 MB 2025-02-14 13:33:19,112 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22163.34 MB 2025-02-14 13:33:19,112 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3097.49 MB 2025-02-14 13:33:19,112 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58808.34 MB 2025-02-14 13:33:19,112 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28372.37 MB 2025-02-14 13:33:19,112 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30435.97 MB 2025-02-14 13:33:19,112 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31029.44 MB 2025-02-14 13:33:19,165 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 13:33:19,165 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 13:33:19,165 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-14 13:33:19,165 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:33:19,165 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22163.34 MB 2025-02-14 13:33:19,165 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20326.70 MB 2025-02-14 13:33:19,165 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1836.64 MB 2025-02-14 13:33:19,165 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28372.37 MB 2025-02-14 13:33:19,165 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36440.11 MB 2025-02-14 13:33:19,165 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8067.74 MB 2025-02-14 13:33:19,165 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31660.06 MB 2025-02-14 13:33:21,098 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 13:33:21,098 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 13:33:21,098 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 13:33:21,098 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:33:21,098 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20326.70 MB 2025-02-14 13:33:21,098 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20857.54 MB 2025-02-14 13:33:21,098 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 13:33:21,098 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36440.11 MB 2025-02-14 13:33:21,098 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26690.45 MB 2025-02-14 13:33:21,098 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9749.66 MB 2025-02-14 13:33:21,098 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24837.68 MB 2025-02-14 13:33:21,112 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 13:33:21,112 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 13:33:21,112 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:33:21,112 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:33:21,112 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20857.54 MB 2025-02-14 13:33:21,112 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22747.08 MB 2025-02-14 13:33:21,112 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 13:33:21,112 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26690.45 MB 2025-02-14 13:33:21,112 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27634.17 MB 2025-02-14 13:33:21,112 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 13:33:21,112 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24164.51 MB 2025-02-14 13:33:21,334 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 13:33:21,334 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 13:33:21,334 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 13:33:21,334 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:33:21,334 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22747.08 MB 2025-02-14 13:33:21,334 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24988.93 MB 2025-02-14 13:33:21,334 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 13:33:21,334 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27634.17 MB 2025-02-14 13:33:21,334 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33296.48 MB 2025-02-14 13:33:21,334 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 13:33:21,334 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30533.22 MB 2025-02-14 13:33:21,335 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 13:33:21,335 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 13:33:21,335 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.24 seconds 2025-02-14 13:33:21,335 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:33:21,335 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20857.54 MB 2025-02-14 13:33:21,335 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24988.93 MB 2025-02-14 13:33:21,335 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 13:33:21,335 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26690.45 MB 2025-02-14 13:33:21,335 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33296.48 MB 2025-02-14 13:33:21,335 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 13:33:21,335 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30533.22 MB 2025-02-14 13:33:21,503 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 13:33:21,503 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 13:33:21,503 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 13:33:21,503 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:33:21,503 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26522.48 MB 2025-02-14 13:33:21,503 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27289.48 MB 2025-02-14 13:33:21,503 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 13:33:21,504 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33296.48 MB 2025-02-14 13:33:21,504 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33711.72 MB 2025-02-14 13:33:21,504 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 13:33:21,504 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27997.27 MB 2025-02-14 13:33:21,522 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 13:33:21,522 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 13:33:21,522 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:33:21,522 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:33:21,522 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27702.37 MB 2025-02-14 13:33:21,522 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27928.63 MB 2025-02-14 13:33:21,522 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.26 MB 2025-02-14 13:33:21,522 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33711.72 MB 2025-02-14 13:33:21,522 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33711.72 MB 2025-02-14 13:33:21,522 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:33:21,522 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28144.51 MB 2025-02-14 13:33:21,523 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 13:33:21,523 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 13:33:21,523 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.13 seconds 2025-02-14 13:33:21,523 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:33:21,523 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16017.28 MB 2025-02-14 13:33:21,523 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28129.70 MB 2025-02-14 13:33:21,523 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12112.42 MB 2025-02-14 13:33:21,523 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58808.34 MB 2025-02-14 13:33:21,523 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33711.72 MB 2025-02-14 13:33:21,523 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25096.62 MB 2025-02-14 13:33:21,523 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28144.51 MB 2025-02-14 13:33:21,794 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 13:33:21,794 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 13:33:21,794 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 13:33:21,794 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:33:21,794 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28129.70 MB 2025-02-14 13:33:21,794 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21021.67 MB 2025-02-14 13:33:21,794 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7108.03 MB 2025-02-14 13:33:21,794 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33711.72 MB 2025-02-14 13:33:21,794 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33711.72 MB 2025-02-14 13:33:21,794 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:33:21,794 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30641.37 MB 2025-02-14 13:33:21,813 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 13:33:21,813 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 13:33:21,819 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 13:33:21,819 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 13:33:21,819 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:33:21,819 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:33:21,819 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21021.67 MB 2025-02-14 13:33:21,819 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29460.69 MB 2025-02-14 13:33:21,819 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 13:33:21,819 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33711.72 MB 2025-02-14 13:33:21,819 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42102.42 MB 2025-02-14 13:33:21,819 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 13:33:21,819 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29460.69 MB 2025-02-14 13:33:21,982 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 13:33:21,984 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:33:21,984 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 13:33:21,985 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:33:21,985 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 13:33:21,990 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 13:33:21,991 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:33:21,991 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 13:33:21,991 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 13:34:38,418 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:34:38,418 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 13:34:38,423 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 13:34:38,427 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:34:38,427 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 300, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 13:34:38,428 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:34:38,428 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 300, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 13:34:43,053 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 13:34:43,053 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 13:34:43,053 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.62 seconds 2025-02-14 13:34:43,053 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:34:43,053 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15059.15 MB 2025-02-14 13:34:43,053 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16120.84 MB 2025-02-14 13:34:43,053 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1061.68 MB 2025-02-14 13:34:43,053 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54687.43 MB 2025-02-14 13:34:43,053 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18798.87 MB 2025-02-14 13:34:43,053 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35888.56 MB 2025-02-14 13:34:43,053 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24984.32 MB 2025-02-14 13:34:43,065 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 13:34:43,065 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 13:34:43,065 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:34:43,065 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:34:43,065 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16120.84 MB 2025-02-14 13:34:43,065 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14661.75 MB 2025-02-14 13:34:43,065 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1459.09 MB 2025-02-14 13:34:43,065 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18798.87 MB 2025-02-14 13:34:43,065 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18798.87 MB 2025-02-14 13:34:43,065 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:34:43,065 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16430.25 MB 2025-02-14 13:34:43,165 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 13:34:43,166 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 13:34:43,166 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 13:34:43,166 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:34:43,166 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14661.75 MB 2025-02-14 13:34:43,166 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14686.97 MB 2025-02-14 13:34:43,166 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 25.21 MB 2025-02-14 13:34:43,166 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18798.87 MB 2025-02-14 13:34:43,166 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18798.87 MB 2025-02-14 13:34:43,166 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:34:43,166 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15874.41 MB 2025-02-14 13:34:43,170 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 13:34:43,170 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 13:34:43,170 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 13:34:43,170 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:34:43,170 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14686.90 MB 2025-02-14 13:34:43,170 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14776.63 MB 2025-02-14 13:34:43,170 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 89.73 MB 2025-02-14 13:34:43,170 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18798.87 MB 2025-02-14 13:34:43,170 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18798.87 MB 2025-02-14 13:34:43,170 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:34:43,170 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14844.39 MB 2025-02-14 13:34:43,185 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 13:34:43,185 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 13:34:43,186 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:34:43,186 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:34:43,186 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14776.63 MB 2025-02-14 13:34:43,186 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14884.58 MB 2025-02-14 13:34:43,186 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 107.95 MB 2025-02-14 13:34:43,186 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18798.87 MB 2025-02-14 13:34:43,186 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18798.87 MB 2025-02-14 13:34:43,186 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:34:43,186 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15148.79 MB 2025-02-14 13:34:43,186 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 13:34:43,186 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 13:34:43,186 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:34:43,186 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:34:43,186 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14686.90 MB 2025-02-14 13:34:43,186 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14884.58 MB 2025-02-14 13:34:43,186 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 197.68 MB 2025-02-14 13:34:43,186 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18798.87 MB 2025-02-14 13:34:43,186 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18798.87 MB 2025-02-14 13:34:43,186 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:34:43,186 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15148.79 MB 2025-02-14 13:34:43,198 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 13:34:43,198 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 13:34:43,198 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:34:43,198 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:34:43,198 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14958.34 MB 2025-02-14 13:34:43,198 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14994.77 MB 2025-02-14 13:34:43,198 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 36.43 MB 2025-02-14 13:34:43,198 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18798.87 MB 2025-02-14 13:34:43,198 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18811.45 MB 2025-02-14 13:34:43,198 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 12.58 MB 2025-02-14 13:34:43,198 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15043.03 MB 2025-02-14 13:34:43,201 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 13:34:43,201 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 13:34:43,201 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 13:34:43,201 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:34:43,201 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15014.39 MB 2025-02-14 13:34:43,201 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15038.98 MB 2025-02-14 13:34:43,201 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 24.59 MB 2025-02-14 13:34:43,201 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18811.45 MB 2025-02-14 13:34:43,201 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18811.45 MB 2025-02-14 13:34:43,201 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:34:43,201 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15038.98 MB 2025-02-14 13:34:43,202 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 13:34:43,202 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 13:34:43,202 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.77 seconds 2025-02-14 13:34:43,202 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:34:43,202 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14013.93 MB 2025-02-14 13:34:43,202 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15084.06 MB 2025-02-14 13:34:43,202 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1070.13 MB 2025-02-14 13:34:43,202 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54687.43 MB 2025-02-14 13:34:43,202 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18811.45 MB 2025-02-14 13:34:43,202 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35875.98 MB 2025-02-14 13:34:43,202 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15084.06 MB 2025-02-14 13:34:43,273 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 13:34:43,273 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 13:34:43,273 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 13:34:43,273 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:34:43,273 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15084.06 MB 2025-02-14 13:34:43,273 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15759.78 MB 2025-02-14 13:34:43,273 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 675.72 MB 2025-02-14 13:34:43,273 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18811.45 MB 2025-02-14 13:34:43,273 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18817.74 MB 2025-02-14 13:34:43,273 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6.29 MB 2025-02-14 13:34:43,273 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15827.34 MB 2025-02-14 13:34:43,278 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 1819, cut from 1821 2025-02-14 13:34:43,279 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-14 13:34:43,281 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 13:34:43,281 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 13:34:43,281 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:34:43,281 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:34:43,281 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14804.82 MB 2025-02-14 13:34:43,281 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16696.31 MB 2025-02-14 13:34:43,281 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1891.50 MB 2025-02-14 13:34:43,281 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18817.74 MB 2025-02-14 13:34:43,281 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18817.74 MB 2025-02-14 13:34:43,281 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:34:43,281 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16696.31 MB 2025-02-14 13:34:43,318 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 1611] 2025-02-14 13:34:43,319 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:34:43,319 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 13:34:43,320 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:34:43,320 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 13:34:43,325 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 13:34:43,326 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:34:43,326 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 13:34:43,326 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-14 13:34:53,198 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:34:53,198 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 13:34:53,203 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 13:34:53,206 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:34:53,206 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1782, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 13:34:53,207 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:34:53,207 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1782, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 13:35:20,884 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 13:35:20,884 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 13:35:20,884 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.67 seconds 2025-02-14 13:35:20,884 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:35:20,884 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25388.04 MB 2025-02-14 13:35:20,884 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31694.44 MB 2025-02-14 13:35:20,884 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6306.40 MB 2025-02-14 13:35:20,884 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33120.32 MB 2025-02-14 13:35:20,884 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35636.90 MB 2025-02-14 13:35:20,884 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2516.58 MB 2025-02-14 13:35:20,884 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40522.53 MB 2025-02-14 13:35:21,006 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 13:35:21,006 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 13:35:21,006 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 13:35:21,006 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:35:21,006 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31694.44 MB 2025-02-14 13:35:21,006 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25043.07 MB 2025-02-14 13:35:21,006 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6651.37 MB 2025-02-14 13:35:21,006 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35636.90 MB 2025-02-14 13:35:21,006 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 59443.77 MB 2025-02-14 13:35:21,006 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 23806.87 MB 2025-02-14 13:35:21,006 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50150.53 MB 2025-02-14 13:35:22,968 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 13:35:22,968 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 13:35:22,968 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.96 seconds 2025-02-14 13:35:22,968 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:35:22,968 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25043.07 MB 2025-02-14 13:35:22,968 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25573.91 MB 2025-02-14 13:35:22,968 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 13:35:22,968 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59443.77 MB 2025-02-14 13:35:22,968 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27590.13 MB 2025-02-14 13:35:22,968 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -31853.64 MB 2025-02-14 13:35:22,968 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29554.28 MB 2025-02-14 13:35:22,982 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 13:35:22,982 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 13:35:22,982 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:35:22,982 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:35:22,982 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25573.91 MB 2025-02-14 13:35:22,982 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27463.44 MB 2025-02-14 13:35:22,982 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 13:35:22,982 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27590.13 MB 2025-02-14 13:35:22,982 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30893.15 MB 2025-02-14 13:35:22,982 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 13:35:22,982 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28880.87 MB 2025-02-14 13:35:23,192 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 13:35:23,192 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 13:35:23,192 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 13:35:23,192 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:35:23,192 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27463.44 MB 2025-02-14 13:35:23,192 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29705.30 MB 2025-02-14 13:35:23,192 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 13:35:23,192 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30893.15 MB 2025-02-14 13:35:23,192 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37499.17 MB 2025-02-14 13:35:23,192 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 13:35:23,192 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35249.58 MB 2025-02-14 13:35:23,193 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 13:35:23,193 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 13:35:23,193 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 13:35:23,193 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:35:23,193 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25573.91 MB 2025-02-14 13:35:23,193 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29705.30 MB 2025-02-14 13:35:23,193 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 13:35:23,193 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27590.13 MB 2025-02-14 13:35:23,193 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37499.17 MB 2025-02-14 13:35:23,193 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9909.04 MB 2025-02-14 13:35:23,193 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35249.58 MB 2025-02-14 13:35:23,365 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 13:35:23,365 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 13:35:23,365 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 13:35:23,365 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:35:23,365 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31238.84 MB 2025-02-14 13:35:23,365 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32005.84 MB 2025-02-14 13:35:23,365 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 13:35:23,365 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37499.17 MB 2025-02-14 13:35:23,365 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37914.41 MB 2025-02-14 13:35:23,365 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 13:35:23,365 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32713.63 MB 2025-02-14 13:35:23,385 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 13:35:23,385 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 13:35:23,385 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:35:23,385 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:35:23,385 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32418.73 MB 2025-02-14 13:35:23,385 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32648.15 MB 2025-02-14 13:35:23,385 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.42 MB 2025-02-14 13:35:23,385 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37914.41 MB 2025-02-14 13:35:23,385 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37914.41 MB 2025-02-14 13:35:23,385 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:35:23,385 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32854.93 MB 2025-02-14 13:35:23,386 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 13:35:23,386 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 13:35:23,386 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.18 seconds 2025-02-14 13:35:23,386 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:35:23,386 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19178.37 MB 2025-02-14 13:35:23,386 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32848.73 MB 2025-02-14 13:35:23,386 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13670.36 MB 2025-02-14 13:35:23,386 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26910.65 MB 2025-02-14 13:35:23,386 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37914.41 MB 2025-02-14 13:35:23,386 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 11003.76 MB 2025-02-14 13:35:23,386 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32854.93 MB 2025-02-14 13:35:23,658 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 13:35:23,658 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 13:35:23,658 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 13:35:23,658 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:35:23,658 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32848.73 MB 2025-02-14 13:35:23,658 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24175.14 MB 2025-02-14 13:35:23,658 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8673.59 MB 2025-02-14 13:35:23,658 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37914.41 MB 2025-02-14 13:35:23,658 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37914.41 MB 2025-02-14 13:35:23,658 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:35:23,658 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35354.25 MB 2025-02-14 13:35:23,675 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8142, cut from 8144 2025-02-14 13:35:23,676 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 13:35:23,682 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 13:35:23,682 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 13:35:23,682 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:35:23,682 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:35:23,682 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24175.14 MB 2025-02-14 13:35:23,682 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32593.30 MB 2025-02-14 13:35:23,682 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8418.15 MB 2025-02-14 13:35:23,682 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37914.41 MB 2025-02-14 13:35:23,682 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46284.14 MB 2025-02-14 13:35:23,682 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8369.73 MB 2025-02-14 13:35:23,682 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32593.30 MB 2025-02-14 13:35:23,845 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7934] 2025-02-14 13:35:23,846 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:35:23,846 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 13:35:23,847 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:35:23,847 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 13:35:23,852 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 13:35:23,853 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:35:23,853 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 13:35:23,853 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 13:35:31,125 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:35:31,125 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 13:35:31,130 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 13:35:31,133 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:35:31,133 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 174, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 13:35:31,134 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:35:31,134 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 174, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 13:35:33,872 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 13:35:33,872 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 13:35:33,872 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.73 seconds 2025-02-14 13:35:33,872 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:35:33,872 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14181.17 MB 2025-02-14 13:35:33,872 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14796.94 MB 2025-02-14 13:35:33,872 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 615.78 MB 2025-02-14 13:35:33,872 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58837.70 MB 2025-02-14 13:35:33,872 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 16909.34 MB 2025-02-14 13:35:33,872 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -41928.36 MB 2025-02-14 13:35:33,872 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23653.34 MB 2025-02-14 13:35:33,886 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 13:35:33,886 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 13:35:33,886 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:35:33,886 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:35:33,886 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14796.94 MB 2025-02-14 13:35:33,886 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15005.41 MB 2025-02-14 13:35:33,886 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 208.47 MB 2025-02-14 13:35:33,886 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 16909.34 MB 2025-02-14 13:35:33,886 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18327.01 MB 2025-02-14 13:35:33,886 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1417.67 MB 2025-02-14 13:35:33,886 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17110.05 MB 2025-02-14 13:35:34,676 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 13:35:34,676 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 13:35:34,676 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.79 seconds 2025-02-14 13:35:34,676 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:35:34,677 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15005.41 MB 2025-02-14 13:35:34,677 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15219.08 MB 2025-02-14 13:35:34,677 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 213.66 MB 2025-02-14 13:35:34,677 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18327.01 MB 2025-02-14 13:35:34,677 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17756.59 MB 2025-02-14 13:35:34,677 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -570.43 MB 2025-02-14 13:35:34,677 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19176.89 MB 2025-02-14 13:35:34,684 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 13:35:34,684 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 13:35:34,684 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 13:35:34,684 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:35:34,684 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15219.01 MB 2025-02-14 13:35:34,684 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15979.36 MB 2025-02-14 13:35:34,684 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 760.35 MB 2025-02-14 13:35:34,684 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17756.59 MB 2025-02-14 13:35:34,684 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17756.59 MB 2025-02-14 13:35:34,684 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:35:34,684 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16549.88 MB 2025-02-14 13:35:34,775 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 13:35:34,775 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 13:35:34,775 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 13:35:34,775 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:35:34,775 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15979.36 MB 2025-02-14 13:35:34,775 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16881.75 MB 2025-02-14 13:35:34,775 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 902.39 MB 2025-02-14 13:35:34,775 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17756.59 MB 2025-02-14 13:35:34,775 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20048.77 MB 2025-02-14 13:35:34,775 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2292.19 MB 2025-02-14 13:35:34,775 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19116.04 MB 2025-02-14 13:35:34,776 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 13:35:34,776 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 13:35:34,776 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 13:35:34,776 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:35:34,776 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15219.01 MB 2025-02-14 13:35:34,776 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16881.75 MB 2025-02-14 13:35:34,776 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1662.74 MB 2025-02-14 13:35:34,776 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17756.59 MB 2025-02-14 13:35:34,776 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20048.77 MB 2025-02-14 13:35:34,776 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2292.19 MB 2025-02-14 13:35:34,776 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19116.04 MB 2025-02-14 13:35:34,880 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 13:35:34,880 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 13:35:34,880 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 13:35:34,880 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:35:34,880 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17499.00 MB 2025-02-14 13:35:34,880 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17809.55 MB 2025-02-14 13:35:34,880 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 310.55 MB 2025-02-14 13:35:34,880 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20048.77 MB 2025-02-14 13:35:34,880 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20214.45 MB 2025-02-14 13:35:34,880 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 165.68 MB 2025-02-14 13:35:34,880 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18101.24 MB 2025-02-14 13:35:34,889 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 13:35:34,889 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 13:35:34,889 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:35:34,889 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:35:34,889 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17975.75 MB 2025-02-14 13:35:34,889 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18204.45 MB 2025-02-14 13:35:34,889 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.70 MB 2025-02-14 13:35:34,890 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20214.45 MB 2025-02-14 13:35:34,890 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20214.45 MB 2025-02-14 13:35:34,890 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:35:34,890 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18228.85 MB 2025-02-14 13:35:34,891 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 13:35:34,891 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 13:35:34,891 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.75 seconds 2025-02-14 13:35:34,891 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:35:34,891 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13574.94 MB 2025-02-14 13:35:34,891 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18405.52 MB 2025-02-14 13:35:34,891 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4830.59 MB 2025-02-14 13:35:34,891 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58837.70 MB 2025-02-14 13:35:34,891 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20214.45 MB 2025-02-14 13:35:34,891 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -38623.25 MB 2025-02-14 13:35:34,891 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18405.52 MB 2025-02-14 13:35:35,159 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 13:35:35,159 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 13:35:35,159 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 13:35:35,159 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:35:35,159 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18405.52 MB 2025-02-14 13:35:35,159 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17453.25 MB 2025-02-14 13:35:35,159 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -952.27 MB 2025-02-14 13:35:35,159 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20214.45 MB 2025-02-14 13:35:35,159 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20214.45 MB 2025-02-14 13:35:35,159 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:35:35,159 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19209.26 MB 2025-02-14 13:35:35,178 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 13:35:35,178 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2,'] 2025-02-14 13:35:35,184 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 13:35:35,185 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 13:35:35,185 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:35:35,185 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:35:35,185 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17453.25 MB 2025-02-14 13:35:35,185 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25892.27 MB 2025-02-14 13:35:35,185 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 13:35:35,185 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20214.45 MB 2025-02-14 13:35:35,185 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30704.40 MB 2025-02-14 13:35:35,185 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 13:35:35,185 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25892.27 MB 2025-02-14 13:35:35,347 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 13:35:35,348 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:35:35,348 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 13:35:35,349 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:35:35,349 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 13:35:35,354 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 13:35:35,355 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:35:35,355 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 13:35:35,355 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2,'] 2025-02-14 13:35:41,024 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:35:41,024 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 13:35:41,032 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 13:35:41,038 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:35:41,039 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 111, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 13:35:41,040 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:35:41,041 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 111, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 13:35:42,813 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 13:35:42,813 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 13:35:42,813 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.77 seconds 2025-02-14 13:35:42,813 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:35:42,813 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13743.21 MB 2025-02-14 13:35:42,813 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14136.03 MB 2025-02-14 13:35:42,813 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 392.82 MB 2025-02-14 13:35:42,813 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43289.41 MB 2025-02-14 13:35:42,813 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17949.52 MB 2025-02-14 13:35:42,813 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25339.89 MB 2025-02-14 13:35:42,814 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22988.09 MB 2025-02-14 13:35:42,816 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 13:35:42,816 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 13:35:42,816 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 13:35:42,816 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:35:42,816 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14136.03 MB 2025-02-14 13:35:42,816 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14326.35 MB 2025-02-14 13:35:42,816 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 190.32 MB 2025-02-14 13:35:42,816 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17949.52 MB 2025-02-14 13:35:42,816 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17949.52 MB 2025-02-14 13:35:42,817 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:35:42,817 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14915.65 MB 2025-02-14 13:35:43,355 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 13:35:43,355 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 13:35:43,355 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.54 seconds 2025-02-14 13:35:43,355 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:35:43,355 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14326.35 MB 2025-02-14 13:35:43,355 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14473.66 MB 2025-02-14 13:35:43,355 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 147.31 MB 2025-02-14 13:35:43,355 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17949.52 MB 2025-02-14 13:35:43,355 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17567.84 MB 2025-02-14 13:35:43,355 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -381.68 MB 2025-02-14 13:35:43,355 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18412.90 MB 2025-02-14 13:35:43,361 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 13:35:43,361 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 13:35:43,361 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 13:35:43,361 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:35:43,361 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14473.60 MB 2025-02-14 13:35:43,361 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14997.82 MB 2025-02-14 13:35:43,361 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 524.22 MB 2025-02-14 13:35:43,361 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17567.84 MB 2025-02-14 13:35:43,361 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17567.84 MB 2025-02-14 13:35:43,361 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:35:43,361 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15391.16 MB 2025-02-14 13:35:43,471 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 13:35:43,471 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 13:35:43,471 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 13:35:43,471 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:35:43,471 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14997.82 MB 2025-02-14 13:35:43,471 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15634.92 MB 2025-02-14 13:35:43,471 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 637.10 MB 2025-02-14 13:35:43,471 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17567.84 MB 2025-02-14 13:35:43,471 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17567.84 MB 2025-02-14 13:35:43,471 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:35:43,471 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17158.47 MB 2025-02-14 13:35:43,472 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 13:35:43,472 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 13:35:43,472 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 13:35:43,472 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:35:43,472 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14473.60 MB 2025-02-14 13:35:43,472 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15634.92 MB 2025-02-14 13:35:43,472 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1161.32 MB 2025-02-14 13:35:43,472 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17567.84 MB 2025-02-14 13:35:43,472 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17567.84 MB 2025-02-14 13:35:43,472 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:35:43,472 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17158.47 MB 2025-02-14 13:35:43,528 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 13:35:43,529 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 13:35:43,529 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-14 13:35:43,529 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:35:43,529 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16249.88 MB 2025-02-14 13:35:43,529 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16517.28 MB 2025-02-14 13:35:43,529 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 267.40 MB 2025-02-14 13:35:43,529 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17567.84 MB 2025-02-14 13:35:43,529 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17737.71 MB 2025-02-14 13:35:43,529 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 169.87 MB 2025-02-14 13:35:43,529 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16713.69 MB 2025-02-14 13:35:43,535 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 13:35:43,535 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 13:35:43,535 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 13:35:43,535 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:35:43,535 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16686.42 MB 2025-02-14 13:35:43,535 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16915.44 MB 2025-02-14 13:35:43,535 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.02 MB 2025-02-14 13:35:43,535 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17737.71 MB 2025-02-14 13:35:43,535 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17737.71 MB 2025-02-14 13:35:43,535 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:35:43,535 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16915.44 MB 2025-02-14 13:35:43,536 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 13:35:43,536 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 13:35:43,536 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.49 seconds 2025-02-14 13:35:43,536 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:35:43,536 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13355.96 MB 2025-02-14 13:35:43,536 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17115.95 MB 2025-02-14 13:35:43,536 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3759.99 MB 2025-02-14 13:35:43,536 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43289.41 MB 2025-02-14 13:35:43,536 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17737.71 MB 2025-02-14 13:35:43,536 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25551.70 MB 2025-02-14 13:35:43,536 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17115.95 MB 2025-02-14 13:35:43,809 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 13:35:43,809 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 13:35:43,809 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 13:35:43,809 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:35:43,809 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17115.95 MB 2025-02-14 13:35:43,809 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20121.50 MB 2025-02-14 13:35:43,809 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3005.55 MB 2025-02-14 13:35:43,809 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17737.71 MB 2025-02-14 13:35:43,809 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21764.24 MB 2025-02-14 13:35:43,809 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4026.53 MB 2025-02-14 13:35:43,809 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20423.30 MB 2025-02-14 13:35:43,827 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8139, cut from 8141 2025-02-14 13:35:43,827 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 13:35:43,834 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 13:35:43,834 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 13:35:43,834 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:35:43,834 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:35:43,834 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20121.50 MB 2025-02-14 13:35:43,834 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28536.45 MB 2025-02-14 13:35:43,834 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8414.95 MB 2025-02-14 13:35:43,834 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21764.24 MB 2025-02-14 13:35:43,834 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32224.84 MB 2025-02-14 13:35:43,834 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10460.59 MB 2025-02-14 13:35:43,834 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28536.45 MB 2025-02-14 13:35:44,002 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7931] 2025-02-14 13:35:44,003 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:35:44,003 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 13:35:44,004 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:35:44,004 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 13:35:44,009 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 13:35:44,010 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:35:44,010 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 13:35:44,011 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 13:36:25,008 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:36:25,008 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 13:36:25,018 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 295 2025-02-14 13:36:25,025 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:36:25,025 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 116, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 13:36:25,027 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:36:25,027 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 116, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 13:36:26,864 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 13:36:26,864 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 13:36:26,864 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.83 seconds 2025-02-14 13:36:26,864 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:36:26,864 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21999.85 MB 2025-02-14 13:36:26,864 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22410.37 MB 2025-02-14 13:36:26,864 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 410.52 MB 2025-02-14 13:36:26,864 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40594.57 MB 2025-02-14 13:36:26,864 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24148.71 MB 2025-02-14 13:36:26,864 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16445.87 MB 2025-02-14 13:36:26,864 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31244.73 MB 2025-02-14 13:36:26,869 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 13:36:26,869 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 13:36:26,869 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 13:36:26,869 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:36:26,869 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22410.37 MB 2025-02-14 13:36:26,869 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22609.26 MB 2025-02-14 13:36:26,869 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 198.89 MB 2025-02-14 13:36:26,869 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24148.71 MB 2025-02-14 13:36:26,869 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24148.71 MB 2025-02-14 13:36:26,870 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:36:26,870 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23225.10 MB 2025-02-14 13:36:27,460 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 13:36:27,460 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 13:36:27,460 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.59 seconds 2025-02-14 13:36:27,460 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:36:27,460 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22609.26 MB 2025-02-14 13:36:27,460 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22763.21 MB 2025-02-14 13:36:27,460 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 153.94 MB 2025-02-14 13:36:27,460 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24148.71 MB 2025-02-14 13:36:27,460 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24148.71 MB 2025-02-14 13:36:27,460 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:36:27,460 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26697.86 MB 2025-02-14 13:36:27,470 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 13:36:27,470 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 13:36:27,470 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:36:27,470 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:36:27,470 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22763.14 MB 2025-02-14 13:36:27,470 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23313.07 MB 2025-02-14 13:36:27,470 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 549.93 MB 2025-02-14 13:36:27,470 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24148.71 MB 2025-02-14 13:36:27,470 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24700.26 MB 2025-02-14 13:36:27,470 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 551.55 MB 2025-02-14 13:36:27,470 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23724.13 MB 2025-02-14 13:36:27,602 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 13:36:27,602 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 13:36:27,602 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 13:36:27,602 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:36:27,602 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23313.07 MB 2025-02-14 13:36:27,602 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23979.51 MB 2025-02-14 13:36:27,602 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 666.44 MB 2025-02-14 13:36:27,603 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24700.26 MB 2025-02-14 13:36:27,603 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26487.03 MB 2025-02-14 13:36:27,603 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1786.77 MB 2025-02-14 13:36:27,603 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25576.29 MB 2025-02-14 13:36:27,604 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 13:36:27,604 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 13:36:27,604 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-14 13:36:27,604 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:36:27,604 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22763.14 MB 2025-02-14 13:36:27,604 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23979.51 MB 2025-02-14 13:36:27,604 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1216.37 MB 2025-02-14 13:36:27,604 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24148.71 MB 2025-02-14 13:36:27,604 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26487.03 MB 2025-02-14 13:36:27,604 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2338.32 MB 2025-02-14 13:36:27,604 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25576.29 MB 2025-02-14 13:36:27,682 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 13:36:27,682 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 13:36:27,682 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 13:36:27,682 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:36:27,683 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24621.89 MB 2025-02-14 13:36:27,683 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25369.43 MB 2025-02-14 13:36:27,683 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 747.54 MB 2025-02-14 13:36:27,683 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26487.03 MB 2025-02-14 13:36:27,683 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26606.57 MB 2025-02-14 13:36:27,683 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 119.54 MB 2025-02-14 13:36:27,683 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25574.69 MB 2025-02-14 13:36:27,696 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 13:36:27,696 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 13:36:27,697 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:36:27,697 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:36:27,697 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25694.43 MB 2025-02-14 13:36:27,697 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25788.96 MB 2025-02-14 13:36:27,697 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 94.52 MB 2025-02-14 13:36:27,697 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26606.57 MB 2025-02-14 13:36:27,697 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26608.66 MB 2025-02-14 13:36:27,697 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-14 13:36:27,697 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25788.96 MB 2025-02-14 13:36:27,699 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 13:36:27,699 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 13:36:27,699 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.67 seconds 2025-02-14 13:36:27,699 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:36:27,699 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21595.70 MB 2025-02-14 13:36:27,699 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25873.63 MB 2025-02-14 13:36:27,699 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4277.93 MB 2025-02-14 13:36:27,699 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40594.57 MB 2025-02-14 13:36:27,699 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26608.66 MB 2025-02-14 13:36:27,699 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13985.91 MB 2025-02-14 13:36:27,699 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25873.63 MB 2025-02-14 13:36:27,861 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 13:36:27,861 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 13:36:27,861 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 13:36:27,861 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:36:27,861 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25873.63 MB 2025-02-14 13:36:27,861 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27142.87 MB 2025-02-14 13:36:27,861 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1269.24 MB 2025-02-14 13:36:27,861 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26608.66 MB 2025-02-14 13:36:27,861 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27856.47 MB 2025-02-14 13:36:27,861 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1247.81 MB 2025-02-14 13:36:27,861 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27269.99 MB 2025-02-14 13:36:27,871 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 3429, cut from 3431 2025-02-14 13:36:27,872 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 13:36:27,876 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 13:36:27,876 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 13:36:27,877 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:36:27,877 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:36:27,877 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27142.87 MB 2025-02-14 13:36:27,877 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30696.19 MB 2025-02-14 13:36:27,877 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3553.33 MB 2025-02-14 13:36:27,877 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27856.47 MB 2025-02-14 13:36:27,877 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32277.27 MB 2025-02-14 13:36:27,877 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4420.80 MB 2025-02-14 13:36:27,877 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30696.19 MB 2025-02-14 13:36:27,988 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 3150] 2025-02-14 13:36:27,990 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:36:27,990 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 308, 128256]), torch.float32, cuda:0] 2025-02-14 13:36:27,992 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:36:27,992 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 309]), torch.int64, cuda:0] 2025-02-14 13:36:28,002 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [296, 308] 2025-02-14 13:36:28,004 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:36:28,004 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 13:36:28,005 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 13:37:26,070 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:37:26,070 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 13:37:26,075 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 13:37:26,079 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:37:26,079 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 852, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 13:37:26,080 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:37:26,080 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 852, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 13:37:39,166 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 13:37:39,167 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 13:37:39,167 - resource_logging.py:150 - __exit__ - DEBUG - Time: 13.08 seconds 2025-02-14 13:37:39,167 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:37:39,167 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18905.58 MB 2025-02-14 13:37:39,167 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21921.28 MB 2025-02-14 13:37:39,167 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3015.70 MB 2025-02-14 13:37:39,167 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35813.06 MB 2025-02-14 13:37:39,167 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27005.03 MB 2025-02-14 13:37:39,167 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8808.04 MB 2025-02-14 13:37:39,167 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30869.17 MB 2025-02-14 13:37:39,222 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 13:37:39,222 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 13:37:39,223 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-14 13:37:39,223 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:37:39,223 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21921.28 MB 2025-02-14 13:37:39,223 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20207.13 MB 2025-02-14 13:37:39,223 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1714.15 MB 2025-02-14 13:37:39,223 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27005.03 MB 2025-02-14 13:37:39,223 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37042.00 MB 2025-02-14 13:37:39,223 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10036.97 MB 2025-02-14 13:37:39,223 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32124.08 MB 2025-02-14 13:37:41,146 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 13:37:41,146 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 13:37:41,146 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 13:37:41,146 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:37:41,146 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20207.13 MB 2025-02-14 13:37:41,146 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20737.97 MB 2025-02-14 13:37:41,146 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 13:37:41,146 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37042.00 MB 2025-02-14 13:37:41,146 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23637.00 MB 2025-02-14 13:37:41,146 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13405.00 MB 2025-02-14 13:37:41,146 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24718.35 MB 2025-02-14 13:37:41,161 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 13:37:41,161 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 13:37:41,161 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:37:41,161 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:37:41,161 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20737.97 MB 2025-02-14 13:37:41,161 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22627.51 MB 2025-02-14 13:37:41,161 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 13:37:41,161 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23637.00 MB 2025-02-14 13:37:41,161 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26940.01 MB 2025-02-14 13:37:41,161 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 13:37:41,161 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24044.94 MB 2025-02-14 13:37:41,368 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 13:37:41,368 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 13:37:41,368 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 13:37:41,368 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:37:41,368 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22627.51 MB 2025-02-14 13:37:41,368 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24869.36 MB 2025-02-14 13:37:41,368 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 13:37:41,368 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26940.01 MB 2025-02-14 13:37:41,368 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33074.18 MB 2025-02-14 13:37:41,368 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 13:37:41,368 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30413.65 MB 2025-02-14 13:37:41,369 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 13:37:41,369 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 13:37:41,369 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 13:37:41,369 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:37:41,369 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20737.97 MB 2025-02-14 13:37:41,369 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24869.36 MB 2025-02-14 13:37:41,369 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 13:37:41,369 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23637.00 MB 2025-02-14 13:37:41,369 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33074.18 MB 2025-02-14 13:37:41,369 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9437.18 MB 2025-02-14 13:37:41,369 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30413.65 MB 2025-02-14 13:37:41,533 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 13:37:41,533 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 13:37:41,533 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 13:37:41,533 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:37:41,533 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26402.91 MB 2025-02-14 13:37:41,533 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27169.91 MB 2025-02-14 13:37:41,533 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 13:37:41,533 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33074.18 MB 2025-02-14 13:37:41,533 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33483.13 MB 2025-02-14 13:37:41,533 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 408.94 MB 2025-02-14 13:37:41,533 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27877.70 MB 2025-02-14 13:37:41,552 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 13:37:41,552 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 13:37:41,552 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:37:41,552 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:37:41,552 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27582.80 MB 2025-02-14 13:37:41,552 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27811.68 MB 2025-02-14 13:37:41,552 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.89 MB 2025-02-14 13:37:41,552 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33483.13 MB 2025-02-14 13:37:41,552 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33483.13 MB 2025-02-14 13:37:41,552 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:37:41,552 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28036.95 MB 2025-02-14 13:37:41,553 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 13:37:41,554 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 13:37:41,554 - resource_logging.py:150 - __exit__ - DEBUG - Time: 15.47 seconds 2025-02-14 13:37:41,554 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:37:41,554 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15937.14 MB 2025-02-14 13:37:41,554 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28012.49 MB 2025-02-14 13:37:41,554 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12075.34 MB 2025-02-14 13:37:41,554 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35813.06 MB 2025-02-14 13:37:41,554 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33483.13 MB 2025-02-14 13:37:41,554 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2329.94 MB 2025-02-14 13:37:41,554 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28036.95 MB 2025-02-14 13:37:41,821 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 13:37:41,821 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 13:37:41,821 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 13:37:41,821 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:37:41,821 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28012.49 MB 2025-02-14 13:37:41,821 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20937.34 MB 2025-02-14 13:37:41,821 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7075.15 MB 2025-02-14 13:37:41,821 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33483.13 MB 2025-02-14 13:37:41,821 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33483.13 MB 2025-02-14 13:37:41,821 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:37:41,821 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30520.77 MB 2025-02-14 13:37:41,851 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8151, cut from 8153 2025-02-14 13:37:41,852 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 13:37:41,858 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 13:37:41,858 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 13:37:41,858 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-14 13:37:41,858 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:37:41,858 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20937.34 MB 2025-02-14 13:37:41,858 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29364.68 MB 2025-02-14 13:37:41,858 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8427.34 MB 2025-02-14 13:37:41,858 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33483.13 MB 2025-02-14 13:37:41,858 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43958.40 MB 2025-02-14 13:37:41,858 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10475.27 MB 2025-02-14 13:37:41,858 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29364.68 MB 2025-02-14 13:37:42,009 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7943] 2025-02-14 13:37:42,010 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:37:42,010 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 13:37:42,011 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:37:42,011 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 13:37:42,016 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 13:37:42,017 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:37:42,017 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 13:37:42,017 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 13:37:50,067 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:37:50,067 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 13:37:50,072 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 13:37:50,075 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:37:50,075 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1406, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 13:37:50,076 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:37:50,076 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1406, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 13:38:11,880 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 13:38:11,881 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 13:38:11,881 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.79 seconds 2025-02-14 13:38:11,881 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:38:11,881 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22765.94 MB 2025-02-14 13:38:11,881 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27742.48 MB 2025-02-14 13:38:11,881 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4976.54 MB 2025-02-14 13:38:11,881 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52338.62 MB 2025-02-14 13:38:11,881 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38646.32 MB 2025-02-14 13:38:11,881 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13692.31 MB 2025-02-14 13:38:11,881 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36540.67 MB 2025-02-14 13:38:11,989 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 13:38:11,989 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 13:38:11,989 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 13:38:11,989 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:38:11,989 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27742.48 MB 2025-02-14 13:38:11,989 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23087.21 MB 2025-02-14 13:38:11,989 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4655.28 MB 2025-02-14 13:38:11,989 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38646.32 MB 2025-02-14 13:38:11,989 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48448.41 MB 2025-02-14 13:38:11,989 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9802.09 MB 2025-02-14 13:38:11,989 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42526.99 MB 2025-02-14 13:38:13,937 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 13:38:13,937 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 13:38:13,937 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.95 seconds 2025-02-14 13:38:13,937 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:38:13,937 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23087.21 MB 2025-02-14 13:38:13,937 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23618.05 MB 2025-02-14 13:38:13,937 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 13:38:13,937 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48448.41 MB 2025-02-14 13:38:13,937 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33669.78 MB 2025-02-14 13:38:13,937 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14778.63 MB 2025-02-14 13:38:13,937 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27597.38 MB 2025-02-14 13:38:13,950 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 13:38:13,950 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 13:38:13,950 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:38:13,950 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:38:13,950 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23618.05 MB 2025-02-14 13:38:13,950 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25507.58 MB 2025-02-14 13:38:13,950 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 13:38:13,950 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33669.78 MB 2025-02-14 13:38:13,950 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33669.78 MB 2025-02-14 13:38:13,950 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:38:13,950 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26925.01 MB 2025-02-14 13:38:14,161 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 13:38:14,161 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 13:38:14,161 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 13:38:14,161 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:38:14,161 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25507.58 MB 2025-02-14 13:38:14,161 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27749.44 MB 2025-02-14 13:38:14,162 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 13:38:14,162 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33669.78 MB 2025-02-14 13:38:14,162 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37444.65 MB 2025-02-14 13:38:14,162 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 13:38:14,162 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33293.72 MB 2025-02-14 13:38:14,162 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 13:38:14,162 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 13:38:14,162 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 13:38:14,162 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:38:14,162 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23618.05 MB 2025-02-14 13:38:14,162 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27749.44 MB 2025-02-14 13:38:14,162 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 13:38:14,162 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33669.78 MB 2025-02-14 13:38:14,162 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37444.65 MB 2025-02-14 13:38:14,162 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 13:38:14,162 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33293.72 MB 2025-02-14 13:38:14,332 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 13:38:14,332 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 13:38:14,332 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 13:38:14,332 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:38:14,332 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29282.98 MB 2025-02-14 13:38:14,332 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30049.98 MB 2025-02-14 13:38:14,332 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 13:38:14,333 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37444.65 MB 2025-02-14 13:38:14,333 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37859.89 MB 2025-02-14 13:38:14,333 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 13:38:14,333 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30757.77 MB 2025-02-14 13:38:14,354 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 13:38:14,354 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 13:38:14,354 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:38:14,354 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:38:14,354 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30462.87 MB 2025-02-14 13:38:14,354 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30691.17 MB 2025-02-14 13:38:14,354 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.30 MB 2025-02-14 13:38:14,354 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37859.89 MB 2025-02-14 13:38:14,354 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37859.89 MB 2025-02-14 13:38:14,354 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:38:14,354 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30937.97 MB 2025-02-14 13:38:14,355 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 13:38:14,355 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 13:38:14,355 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.28 seconds 2025-02-14 13:38:14,355 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:38:14,355 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17867.32 MB 2025-02-14 13:38:14,355 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30891.38 MB 2025-02-14 13:38:14,355 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13024.06 MB 2025-02-14 13:38:14,355 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52338.62 MB 2025-02-14 13:38:14,355 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37859.89 MB 2025-02-14 13:38:14,355 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14478.74 MB 2025-02-14 13:38:14,355 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30937.97 MB 2025-02-14 13:38:14,627 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 13:38:14,627 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 13:38:14,627 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 13:38:14,627 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:38:14,627 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30891.38 MB 2025-02-14 13:38:14,627 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22858.38 MB 2025-02-14 13:38:14,627 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8033.00 MB 2025-02-14 13:38:14,627 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37859.89 MB 2025-02-14 13:38:14,627 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37859.89 MB 2025-02-14 13:38:14,627 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:38:14,627 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33392.30 MB 2025-02-14 13:38:14,644 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8127, cut from 8129 2025-02-14 13:38:14,645 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-14 13:38:14,651 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 13:38:14,651 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 13:38:14,651 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:38:14,651 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:38:14,651 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22858.38 MB 2025-02-14 13:38:14,651 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31260.94 MB 2025-02-14 13:38:14,651 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8402.56 MB 2025-02-14 13:38:14,651 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37859.89 MB 2025-02-14 13:38:14,651 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42037.41 MB 2025-02-14 13:38:14,651 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4177.53 MB 2025-02-14 13:38:14,651 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31260.94 MB 2025-02-14 13:38:14,812 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7919] 2025-02-14 13:38:14,814 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:38:14,814 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 13:38:14,815 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:38:14,815 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 13:38:14,820 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 13:38:14,821 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:38:14,821 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 13:38:14,821 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-14 13:39:32,147 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:39:32,147 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 13:39:32,155 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 13:39:32,163 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:39:32,163 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 155, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 13:39:32,165 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:39:32,165 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 155, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 13:39:34,669 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 13:39:34,669 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 13:39:34,669 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.50 seconds 2025-02-14 13:39:34,669 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:39:34,669 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14048.77 MB 2025-02-14 13:39:34,669 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14597.31 MB 2025-02-14 13:39:34,669 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 548.54 MB 2025-02-14 13:39:34,669 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50392.47 MB 2025-02-14 13:39:34,669 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22043.16 MB 2025-02-14 13:39:34,669 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28349.30 MB 2025-02-14 13:39:34,669 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23520.14 MB 2025-02-14 13:39:34,685 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 13:39:34,685 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 13:39:34,685 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:39:34,685 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:39:34,685 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14597.31 MB 2025-02-14 13:39:34,685 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14820.93 MB 2025-02-14 13:39:34,685 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 223.63 MB 2025-02-14 13:39:34,685 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22043.16 MB 2025-02-14 13:39:34,685 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22043.16 MB 2025-02-14 13:39:34,685 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:39:34,685 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16729.16 MB 2025-02-14 13:39:35,406 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 13:39:35,406 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 13:39:35,407 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.72 seconds 2025-02-14 13:39:35,407 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:39:35,407 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14820.93 MB 2025-02-14 13:39:35,407 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15018.67 MB 2025-02-14 13:39:35,407 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 197.74 MB 2025-02-14 13:39:35,407 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22043.16 MB 2025-02-14 13:39:35,407 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22043.16 MB 2025-02-14 13:39:35,407 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:39:35,407 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18991.37 MB 2025-02-14 13:39:35,414 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 13:39:35,414 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 13:39:35,414 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 13:39:35,414 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:39:35,414 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15018.61 MB 2025-02-14 13:39:35,414 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15722.29 MB 2025-02-14 13:39:35,414 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 703.68 MB 2025-02-14 13:39:35,414 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22043.16 MB 2025-02-14 13:39:35,414 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22043.16 MB 2025-02-14 13:39:35,414 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:39:35,414 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16250.29 MB 2025-02-14 13:39:35,496 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 13:39:35,496 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 13:39:35,496 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 13:39:35,496 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:39:35,496 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15722.29 MB 2025-02-14 13:39:35,496 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16557.42 MB 2025-02-14 13:39:35,496 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 835.13 MB 2025-02-14 13:39:35,496 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22043.16 MB 2025-02-14 13:39:35,496 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22043.16 MB 2025-02-14 13:39:35,496 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:39:35,496 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18622.62 MB 2025-02-14 13:39:35,497 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 13:39:35,497 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 13:39:35,497 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 13:39:35,497 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:39:35,497 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15018.61 MB 2025-02-14 13:39:35,497 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16557.42 MB 2025-02-14 13:39:35,497 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1538.81 MB 2025-02-14 13:39:35,497 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22043.16 MB 2025-02-14 13:39:35,497 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22043.16 MB 2025-02-14 13:39:35,497 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:39:35,497 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18622.62 MB 2025-02-14 13:39:35,563 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 13:39:35,563 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 13:39:35,564 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 13:39:35,564 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:39:35,564 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17128.66 MB 2025-02-14 13:39:35,564 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17414.37 MB 2025-02-14 13:39:35,564 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 285.71 MB 2025-02-14 13:39:35,564 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22043.16 MB 2025-02-14 13:39:35,564 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22196.26 MB 2025-02-14 13:39:35,564 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 153.09 MB 2025-02-14 13:39:35,564 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17688.78 MB 2025-02-14 13:39:35,573 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 13:39:35,573 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 13:39:35,573 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:39:35,573 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:39:35,573 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17568.18 MB 2025-02-14 13:39:35,573 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17784.84 MB 2025-02-14 13:39:35,573 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 216.66 MB 2025-02-14 13:39:35,573 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22196.26 MB 2025-02-14 13:39:35,573 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22196.26 MB 2025-02-14 13:39:35,573 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:39:35,573 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17800.07 MB 2025-02-14 13:39:35,574 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 13:39:35,574 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 13:39:35,574 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.41 seconds 2025-02-14 13:39:35,574 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:39:35,574 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13508.74 MB 2025-02-14 13:39:35,574 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17985.84 MB 2025-02-14 13:39:35,574 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4477.10 MB 2025-02-14 13:39:35,574 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50392.47 MB 2025-02-14 13:39:35,574 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22196.26 MB 2025-02-14 13:39:35,574 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28196.21 MB 2025-02-14 13:39:35,574 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17985.84 MB 2025-02-14 13:39:35,840 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 13:39:35,840 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 13:39:35,840 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 13:39:35,840 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:39:35,840 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17985.84 MB 2025-02-14 13:39:35,840 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17327.44 MB 2025-02-14 13:39:35,840 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -658.39 MB 2025-02-14 13:39:35,840 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22196.26 MB 2025-02-14 13:39:35,840 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22196.26 MB 2025-02-14 13:39:35,840 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:39:35,840 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18990.13 MB 2025-02-14 13:39:35,857 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8159, cut from 8161 2025-02-14 13:39:35,858 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2.'] 2025-02-14 13:39:35,864 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 13:39:35,864 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 13:39:35,864 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:39:35,864 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:39:35,864 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17327.44 MB 2025-02-14 13:39:35,864 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25763.07 MB 2025-02-14 13:39:35,864 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8435.62 MB 2025-02-14 13:39:35,864 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22196.26 MB 2025-02-14 13:39:35,864 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30584.86 MB 2025-02-14 13:39:35,864 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8388.61 MB 2025-02-14 13:39:35,864 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25763.07 MB 2025-02-14 13:39:36,027 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7951] 2025-02-14 13:39:36,028 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:39:36,028 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 13:39:36,029 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:39:36,029 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 13:39:36,034 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 13:39:36,035 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:39:36,035 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 13:39:36,035 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2.'] 2025-02-14 13:40:20,593 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:40:20,593 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 13:40:20,598 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 13:40:20,602 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:40:20,602 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1838, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 13:40:20,603 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:40:20,603 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1838, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 13:40:48,860 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 13:40:48,860 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 13:40:48,860 - resource_logging.py:150 - __exit__ - DEBUG - Time: 28.25 seconds 2025-02-14 13:40:48,860 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:40:48,860 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25776.18 MB 2025-02-14 13:40:48,860 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32281.55 MB 2025-02-14 13:40:48,860 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6505.37 MB 2025-02-14 13:40:48,860 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38973.47 MB 2025-02-14 13:40:48,860 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40185.63 MB 2025-02-14 13:40:48,860 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1212.15 MB 2025-02-14 13:40:48,860 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41137.16 MB 2025-02-14 13:40:48,965 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 13:40:48,965 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 13:40:48,965 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 13:40:48,965 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:40:48,965 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32281.55 MB 2025-02-14 13:40:48,965 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25333.04 MB 2025-02-14 13:40:48,965 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6948.51 MB 2025-02-14 13:40:48,965 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40185.63 MB 2025-02-14 13:40:48,965 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56809.75 MB 2025-02-14 13:40:48,965 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 16624.12 MB 2025-02-14 13:40:48,965 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48443.61 MB 2025-02-14 13:40:50,890 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 13:40:50,890 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 13:40:50,890 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 13:40:50,890 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:40:50,890 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25333.04 MB 2025-02-14 13:40:50,890 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25863.88 MB 2025-02-14 13:40:50,890 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 13:40:50,890 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56809.75 MB 2025-02-14 13:40:50,890 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30901.53 MB 2025-02-14 13:40:50,890 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25908.22 MB 2025-02-14 13:40:50,890 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29843.21 MB 2025-02-14 13:40:50,907 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 13:40:50,907 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 13:40:50,907 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:40:50,907 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:40:50,907 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25863.88 MB 2025-02-14 13:40:50,907 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27753.41 MB 2025-02-14 13:40:50,907 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 13:40:50,907 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30901.53 MB 2025-02-14 13:40:50,907 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32788.97 MB 2025-02-14 13:40:50,907 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 13:40:50,907 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29170.84 MB 2025-02-14 13:40:51,118 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 13:40:51,118 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 13:40:51,118 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 13:40:51,118 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:40:51,118 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27753.41 MB 2025-02-14 13:40:51,118 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29995.27 MB 2025-02-14 13:40:51,118 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 13:40:51,118 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32788.97 MB 2025-02-14 13:40:51,118 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38451.28 MB 2025-02-14 13:40:51,118 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 13:40:51,118 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35539.55 MB 2025-02-14 13:40:51,119 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 13:40:51,119 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 13:40:51,119 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 13:40:51,119 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:40:51,119 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25863.88 MB 2025-02-14 13:40:51,119 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29995.27 MB 2025-02-14 13:40:51,119 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 13:40:51,119 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30901.53 MB 2025-02-14 13:40:51,119 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38451.28 MB 2025-02-14 13:40:51,119 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-14 13:40:51,119 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35539.55 MB 2025-02-14 13:40:51,291 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 13:40:51,291 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 13:40:51,291 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 13:40:51,291 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:40:51,291 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31528.81 MB 2025-02-14 13:40:51,291 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32295.81 MB 2025-02-14 13:40:51,291 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 13:40:51,291 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38451.28 MB 2025-02-14 13:40:51,291 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38868.62 MB 2025-02-14 13:40:51,291 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 13:40:51,291 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33003.60 MB 2025-02-14 13:40:51,311 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 13:40:51,311 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 13:40:51,311 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:40:51,311 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:40:51,311 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32708.70 MB 2025-02-14 13:40:51,311 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32937.03 MB 2025-02-14 13:40:51,311 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.32 MB 2025-02-14 13:40:51,311 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38868.62 MB 2025-02-14 13:40:51,311 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38868.62 MB 2025-02-14 13:40:51,311 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:40:51,311 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33182.45 MB 2025-02-14 13:40:51,312 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 13:40:51,312 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 13:40:51,312 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.71 seconds 2025-02-14 13:40:51,312 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:40:51,312 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19372.45 MB 2025-02-14 13:40:51,312 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33137.90 MB 2025-02-14 13:40:51,312 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13765.46 MB 2025-02-14 13:40:51,312 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38973.47 MB 2025-02-14 13:40:51,312 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38868.62 MB 2025-02-14 13:40:51,312 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -104.86 MB 2025-02-14 13:40:51,312 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33182.45 MB 2025-02-14 13:40:51,582 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 13:40:51,582 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 13:40:51,582 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 13:40:51,582 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:40:51,582 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33137.90 MB 2025-02-14 13:40:51,582 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24373.79 MB 2025-02-14 13:40:51,582 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8764.11 MB 2025-02-14 13:40:51,582 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38868.62 MB 2025-02-14 13:40:51,582 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38868.62 MB 2025-02-14 13:40:51,582 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:40:51,582 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35647.11 MB 2025-02-14 13:40:51,600 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8154, cut from 8156 2025-02-14 13:40:51,600 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 13:40:51,606 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 13:40:51,606 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 13:40:51,606 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:40:51,606 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:40:51,606 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24373.79 MB 2025-02-14 13:40:51,606 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32804.86 MB 2025-02-14 13:40:51,607 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8431.07 MB 2025-02-14 13:40:51,607 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38868.62 MB 2025-02-14 13:40:51,607 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47250.93 MB 2025-02-14 13:40:51,607 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8382.32 MB 2025-02-14 13:40:51,607 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32804.86 MB 2025-02-14 13:40:51,770 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7946] 2025-02-14 13:40:51,771 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:40:51,771 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 13:40:51,772 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:40:51,772 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 13:40:51,777 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 13:40:51,778 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:40:51,778 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 13:40:51,778 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 13:42:06,489 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:42:06,490 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 13:42:06,495 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 13:42:06,499 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:42:06,499 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1201, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 13:42:06,500 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:42:06,500 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1201, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 13:42:25,042 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 13:42:25,043 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 13:42:25,043 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.54 seconds 2025-02-14 13:42:25,043 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:42:25,043 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21337.47 MB 2025-02-14 13:42:25,043 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25588.39 MB 2025-02-14 13:42:25,043 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4250.93 MB 2025-02-14 13:42:25,043 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59823.36 MB 2025-02-14 13:42:25,043 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29540.48 MB 2025-02-14 13:42:25,043 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30282.87 MB 2025-02-14 13:42:25,043 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34433.52 MB 2025-02-14 13:42:25,129 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 13:42:25,129 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 13:42:25,129 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 13:42:25,129 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:42:25,129 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25588.39 MB 2025-02-14 13:42:25,129 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22021.48 MB 2025-02-14 13:42:25,129 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3566.92 MB 2025-02-14 13:42:25,129 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29540.48 MB 2025-02-14 13:42:25,129 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45552.24 MB 2025-02-14 13:42:25,129 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 16011.76 MB 2025-02-14 13:42:25,129 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38203.22 MB 2025-02-14 13:42:27,058 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 13:42:27,059 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 13:42:27,059 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 13:42:27,059 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:42:27,059 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22021.48 MB 2025-02-14 13:42:27,059 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22552.32 MB 2025-02-14 13:42:27,059 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 13:42:27,059 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45552.24 MB 2025-02-14 13:42:27,059 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26705.13 MB 2025-02-14 13:42:27,059 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18847.11 MB 2025-02-14 13:42:27,059 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26532.69 MB 2025-02-14 13:42:27,072 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 13:42:27,072 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 13:42:27,072 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:42:27,072 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:42:27,072 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22552.32 MB 2025-02-14 13:42:27,072 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24441.85 MB 2025-02-14 13:42:27,072 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 13:42:27,072 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26705.13 MB 2025-02-14 13:42:27,072 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28592.57 MB 2025-02-14 13:42:27,072 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 13:42:27,072 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25859.28 MB 2025-02-14 13:42:27,282 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 13:42:27,282 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 13:42:27,282 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 13:42:27,282 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:42:27,282 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24441.85 MB 2025-02-14 13:42:27,282 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26683.71 MB 2025-02-14 13:42:27,282 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 13:42:27,282 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28592.57 MB 2025-02-14 13:42:27,282 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34254.88 MB 2025-02-14 13:42:27,282 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 13:42:27,282 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32227.99 MB 2025-02-14 13:42:27,283 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 13:42:27,283 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 13:42:27,283 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 13:42:27,283 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:42:27,283 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22552.32 MB 2025-02-14 13:42:27,283 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26683.71 MB 2025-02-14 13:42:27,283 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 13:42:27,283 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26705.13 MB 2025-02-14 13:42:27,283 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34254.88 MB 2025-02-14 13:42:27,283 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-14 13:42:27,283 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32227.99 MB 2025-02-14 13:42:27,455 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 13:42:27,455 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 13:42:27,455 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 13:42:27,455 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:42:27,455 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28217.25 MB 2025-02-14 13:42:27,455 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28984.25 MB 2025-02-14 13:42:27,455 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 13:42:27,455 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34254.88 MB 2025-02-14 13:42:27,455 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34670.12 MB 2025-02-14 13:42:27,455 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 13:42:27,455 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29692.04 MB 2025-02-14 13:42:27,474 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 13:42:27,474 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 13:42:27,474 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:42:27,474 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:42:27,474 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29397.14 MB 2025-02-14 13:42:27,474 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29625.58 MB 2025-02-14 13:42:27,474 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.44 MB 2025-02-14 13:42:27,474 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34670.12 MB 2025-02-14 13:42:27,474 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34670.12 MB 2025-02-14 13:42:27,474 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:42:27,474 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29863.37 MB 2025-02-14 13:42:27,475 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 13:42:27,475 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 13:42:27,475 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.97 seconds 2025-02-14 13:42:27,475 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:42:27,476 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17153.09 MB 2025-02-14 13:42:27,476 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29825.54 MB 2025-02-14 13:42:27,476 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12672.46 MB 2025-02-14 13:42:27,476 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59823.36 MB 2025-02-14 13:42:27,476 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34670.12 MB 2025-02-14 13:42:27,476 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25153.24 MB 2025-02-14 13:42:27,476 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29863.37 MB 2025-02-14 13:42:27,743 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 13:42:27,743 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 13:42:27,743 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 13:42:27,743 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:42:27,743 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29825.54 MB 2025-02-14 13:42:27,743 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22140.01 MB 2025-02-14 13:42:27,743 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7685.54 MB 2025-02-14 13:42:27,743 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34670.12 MB 2025-02-14 13:42:27,743 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34670.12 MB 2025-02-14 13:42:27,743 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:42:27,743 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32323.39 MB 2025-02-14 13:42:27,762 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8117, cut from 8119 2025-02-14 13:42:27,762 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 13:42:27,768 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 13:42:27,768 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 13:42:27,768 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:42:27,768 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:42:27,768 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22140.01 MB 2025-02-14 13:42:27,768 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30532.60 MB 2025-02-14 13:42:27,768 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8392.59 MB 2025-02-14 13:42:27,768 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34670.12 MB 2025-02-14 13:42:27,768 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43014.68 MB 2025-02-14 13:42:27,768 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8344.57 MB 2025-02-14 13:42:27,768 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30532.60 MB 2025-02-14 13:42:27,929 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7909] 2025-02-14 13:42:27,931 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:42:27,931 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 13:42:27,932 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:42:27,932 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 13:42:27,937 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 13:42:27,938 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:42:27,938 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 13:42:27,938 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 13:44:10,757 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:44:10,757 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 13:44:10,762 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 13:44:10,766 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:44:10,766 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1501, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 13:44:10,767 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:44:10,767 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1501, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 13:44:33,783 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 13:44:33,783 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 13:44:33,783 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.01 seconds 2025-02-14 13:44:33,783 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:44:33,783 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23427.91 MB 2025-02-14 13:44:33,783 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28740.00 MB 2025-02-14 13:44:33,783 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5312.09 MB 2025-02-14 13:44:33,783 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55530.49 MB 2025-02-14 13:44:33,783 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38908.46 MB 2025-02-14 13:44:33,783 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16622.03 MB 2025-02-14 13:44:33,783 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37656.43 MB 2025-02-14 13:44:33,870 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 13:44:33,870 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 13:44:33,870 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 13:44:33,870 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:44:33,870 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28740.00 MB 2025-02-14 13:44:33,870 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23581.08 MB 2025-02-14 13:44:33,870 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5158.92 MB 2025-02-14 13:44:33,870 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38908.46 MB 2025-02-14 13:44:33,870 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49413.10 MB 2025-02-14 13:44:33,870 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10504.63 MB 2025-02-14 13:44:33,870 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44165.46 MB 2025-02-14 13:44:35,786 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 13:44:35,786 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 13:44:35,786 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 13:44:35,786 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:44:35,786 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23581.08 MB 2025-02-14 13:44:35,786 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24111.92 MB 2025-02-14 13:44:35,786 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 13:44:35,786 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49413.10 MB 2025-02-14 13:44:35,786 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33596.38 MB 2025-02-14 13:44:35,786 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15816.72 MB 2025-02-14 13:44:35,786 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28091.26 MB 2025-02-14 13:44:35,800 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 13:44:35,800 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 13:44:35,800 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:44:35,800 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:44:35,800 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24111.92 MB 2025-02-14 13:44:35,800 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26001.46 MB 2025-02-14 13:44:35,800 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 13:44:35,800 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33596.38 MB 2025-02-14 13:44:35,800 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33596.38 MB 2025-02-14 13:44:35,800 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:44:35,800 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27418.89 MB 2025-02-14 13:44:36,008 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 13:44:36,008 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 13:44:36,008 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 13:44:36,008 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:44:36,008 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26001.46 MB 2025-02-14 13:44:36,008 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28243.31 MB 2025-02-14 13:44:36,008 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 13:44:36,008 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33596.38 MB 2025-02-14 13:44:36,008 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37371.25 MB 2025-02-14 13:44:36,008 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 13:44:36,008 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33787.59 MB 2025-02-14 13:44:36,009 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 13:44:36,009 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 13:44:36,009 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 13:44:36,009 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:44:36,009 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24111.92 MB 2025-02-14 13:44:36,009 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28243.31 MB 2025-02-14 13:44:36,009 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 13:44:36,009 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33596.38 MB 2025-02-14 13:44:36,009 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37371.25 MB 2025-02-14 13:44:36,009 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 13:44:36,009 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33787.59 MB 2025-02-14 13:44:36,174 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 13:44:36,174 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 13:44:36,174 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 13:44:36,174 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:44:36,174 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29776.86 MB 2025-02-14 13:44:36,174 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30543.86 MB 2025-02-14 13:44:36,174 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 13:44:36,174 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37371.25 MB 2025-02-14 13:44:36,174 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37786.48 MB 2025-02-14 13:44:36,174 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 13:44:36,174 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31251.65 MB 2025-02-14 13:44:36,193 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 13:44:36,193 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 13:44:36,193 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:44:36,193 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:44:36,193 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30956.75 MB 2025-02-14 13:44:36,193 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31185.71 MB 2025-02-14 13:44:36,193 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.96 MB 2025-02-14 13:44:36,193 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37786.48 MB 2025-02-14 13:44:36,193 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37786.48 MB 2025-02-14 13:44:36,193 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:44:36,193 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31400.59 MB 2025-02-14 13:44:36,194 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 13:44:36,194 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 13:44:36,194 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.43 seconds 2025-02-14 13:44:36,194 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:44:36,194 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18198.31 MB 2025-02-14 13:44:36,194 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31386.51 MB 2025-02-14 13:44:36,194 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13188.20 MB 2025-02-14 13:44:36,194 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55530.49 MB 2025-02-14 13:44:36,194 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37786.48 MB 2025-02-14 13:44:36,194 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17744.00 MB 2025-02-14 13:44:36,194 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31400.59 MB 2025-02-14 13:44:36,475 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 13:44:36,475 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 13:44:36,475 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.28 seconds 2025-02-14 13:44:36,475 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:44:36,475 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31386.51 MB 2025-02-14 13:44:36,475 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23199.05 MB 2025-02-14 13:44:36,475 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8187.46 MB 2025-02-14 13:44:36,475 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37786.48 MB 2025-02-14 13:44:36,475 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37786.48 MB 2025-02-14 13:44:36,475 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:44:36,476 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33894.80 MB 2025-02-14 13:44:36,494 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8151, cut from 8153 2025-02-14 13:44:36,495 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 13:44:36,502 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 13:44:36,502 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 13:44:36,502 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:44:36,502 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:44:36,502 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23199.05 MB 2025-02-14 13:44:36,502 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31626.39 MB 2025-02-14 13:44:36,502 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8427.34 MB 2025-02-14 13:44:36,502 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37786.48 MB 2025-02-14 13:44:36,502 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46166.70 MB 2025-02-14 13:44:36,502 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8380.22 MB 2025-02-14 13:44:36,502 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31626.39 MB 2025-02-14 13:44:36,746 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7943] 2025-02-14 13:44:36,749 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:44:36,749 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 13:44:36,751 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:44:36,751 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 13:44:36,758 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 13:44:36,761 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:44:36,761 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 13:44:36,761 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 13:44:47,531 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:44:47,531 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 13:44:47,536 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 13:44:47,541 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:44:47,541 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2564, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 13:44:47,543 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:44:47,543 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2564, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 13:45:27,703 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 13:45:27,703 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 13:45:27,703 - resource_logging.py:150 - __exit__ - DEBUG - Time: 40.15 seconds 2025-02-14 13:45:27,703 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:45:27,703 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30835.07 MB 2025-02-14 13:45:27,703 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39909.45 MB 2025-02-14 13:45:27,703 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 9074.38 MB 2025-02-14 13:45:27,703 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 72418.85 MB 2025-02-14 13:45:27,703 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43855.64 MB 2025-02-14 13:45:27,703 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28563.21 MB 2025-02-14 13:45:27,703 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48984.33 MB 2025-02-14 13:45:27,876 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 13:45:27,876 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 13:45:27,876 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 13:45:27,876 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:45:27,876 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39909.45 MB 2025-02-14 13:45:27,876 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29107.29 MB 2025-02-14 13:45:27,876 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10802.16 MB 2025-02-14 13:45:27,876 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43855.64 MB 2025-02-14 13:45:27,876 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 78653.69 MB 2025-02-14 13:45:27,876 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 34798.04 MB 2025-02-14 13:45:27,876 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 66522.36 MB 2025-02-14 13:45:29,863 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 13:45:29,863 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 13:45:29,863 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.99 seconds 2025-02-14 13:45:29,863 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:45:29,863 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29107.29 MB 2025-02-14 13:45:29,863 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29638.13 MB 2025-02-14 13:45:29,863 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 13:45:29,863 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 78653.69 MB 2025-02-14 13:45:29,863 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31658.61 MB 2025-02-14 13:45:29,863 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -46995.08 MB 2025-02-14 13:45:29,863 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33618.50 MB 2025-02-14 13:45:29,877 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 13:45:29,877 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 13:45:29,877 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:45:29,877 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:45:29,877 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29638.13 MB 2025-02-14 13:45:29,877 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31527.66 MB 2025-02-14 13:45:29,877 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 13:45:29,877 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31658.61 MB 2025-02-14 13:45:29,877 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34961.62 MB 2025-02-14 13:45:29,877 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 13:45:29,877 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32945.09 MB 2025-02-14 13:45:30,088 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 13:45:30,088 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 13:45:30,088 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 13:45:30,088 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:45:30,088 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31527.66 MB 2025-02-14 13:45:30,088 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33769.52 MB 2025-02-14 13:45:30,088 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 13:45:30,088 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34961.62 MB 2025-02-14 13:45:30,088 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41567.65 MB 2025-02-14 13:45:30,088 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 13:45:30,088 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39313.80 MB 2025-02-14 13:45:30,089 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 13:45:30,089 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 13:45:30,089 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 13:45:30,089 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:45:30,089 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29638.13 MB 2025-02-14 13:45:30,089 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33769.52 MB 2025-02-14 13:45:30,089 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 13:45:30,089 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31658.61 MB 2025-02-14 13:45:30,089 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41567.65 MB 2025-02-14 13:45:30,089 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9909.04 MB 2025-02-14 13:45:30,089 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39313.80 MB 2025-02-14 13:45:30,260 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 13:45:30,260 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 13:45:30,260 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 13:45:30,260 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:45:30,260 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35303.06 MB 2025-02-14 13:45:30,260 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36070.06 MB 2025-02-14 13:45:30,260 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 13:45:30,260 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41567.65 MB 2025-02-14 13:45:30,260 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41982.89 MB 2025-02-14 13:45:30,260 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 13:45:30,260 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36777.85 MB 2025-02-14 13:45:30,279 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 13:45:30,279 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 13:45:30,279 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:45:30,279 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:45:30,279 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36482.95 MB 2025-02-14 13:45:30,279 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36711.07 MB 2025-02-14 13:45:30,279 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.11 MB 2025-02-14 13:45:30,279 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41982.89 MB 2025-02-14 13:45:30,279 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41982.89 MB 2025-02-14 13:45:30,279 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:45:30,279 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36918.44 MB 2025-02-14 13:45:30,280 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 13:45:30,280 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 13:45:30,280 - resource_logging.py:150 - __exit__ - DEBUG - Time: 42.73 seconds 2025-02-14 13:45:30,280 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:45:30,280 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21901.89 MB 2025-02-14 13:45:30,280 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36911.72 MB 2025-02-14 13:45:30,280 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 15009.83 MB 2025-02-14 13:45:30,280 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63482.89 MB 2025-02-14 13:45:30,280 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41982.89 MB 2025-02-14 13:45:30,280 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21500.00 MB 2025-02-14 13:45:30,280 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36918.44 MB 2025-02-14 13:45:30,552 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 13:45:30,552 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 13:45:30,552 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 13:45:30,552 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:45:30,552 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36911.72 MB 2025-02-14 13:45:30,552 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26899.18 MB 2025-02-14 13:45:30,552 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10012.55 MB 2025-02-14 13:45:30,552 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41982.89 MB 2025-02-14 13:45:30,552 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41982.89 MB 2025-02-14 13:45:30,552 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:45:30,552 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39418.17 MB 2025-02-14 13:45:30,570 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8145, cut from 8147 2025-02-14 13:45:30,571 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 13:45:30,576 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 13:45:30,576 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 13:45:30,576 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:45:30,576 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:45:30,576 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26899.18 MB 2025-02-14 13:45:30,576 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35320.47 MB 2025-02-14 13:45:30,576 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8421.30 MB 2025-02-14 13:45:30,576 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41982.89 MB 2025-02-14 13:45:30,576 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46168.80 MB 2025-02-14 13:45:30,577 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4185.92 MB 2025-02-14 13:45:30,577 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35320.47 MB 2025-02-14 13:45:30,739 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7937] 2025-02-14 13:45:30,740 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:45:30,740 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 13:45:30,741 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:45:30,741 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 13:45:30,746 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 13:45:30,747 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:45:30,747 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 13:45:30,747 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 13:46:30,672 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:46:30,672 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 13:46:30,677 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 13:46:30,681 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:46:30,681 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 197, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 13:46:30,682 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:46:30,682 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 197, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 13:46:33,763 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 13:46:33,763 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 13:46:33,763 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.08 seconds 2025-02-14 13:46:33,763 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:46:33,763 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14341.43 MB 2025-02-14 13:46:33,763 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15038.61 MB 2025-02-14 13:46:33,763 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 697.17 MB 2025-02-14 13:46:33,763 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54540.63 MB 2025-02-14 13:46:33,763 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 16907.24 MB 2025-02-14 13:46:33,763 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -37633.39 MB 2025-02-14 13:46:33,763 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24040.10 MB 2025-02-14 13:46:33,779 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 13:46:33,779 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 13:46:33,779 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:46:33,779 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:46:33,779 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15038.61 MB 2025-02-14 13:46:33,779 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15249.97 MB 2025-02-14 13:46:33,779 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 211.36 MB 2025-02-14 13:46:33,779 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 16907.24 MB 2025-02-14 13:46:33,779 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18486.39 MB 2025-02-14 13:46:33,779 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1579.16 MB 2025-02-14 13:46:33,779 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17570.61 MB 2025-02-14 13:46:34,634 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 13:46:34,634 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 13:46:34,634 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.85 seconds 2025-02-14 13:46:34,634 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:46:34,634 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15249.97 MB 2025-02-14 13:46:34,634 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15487.52 MB 2025-02-14 13:46:34,634 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 237.55 MB 2025-02-14 13:46:34,634 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18486.39 MB 2025-02-14 13:46:34,634 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17863.54 MB 2025-02-14 13:46:34,634 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -622.85 MB 2025-02-14 13:46:34,634 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19421.45 MB 2025-02-14 13:46:34,643 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 13:46:34,643 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 13:46:34,643 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:46:34,643 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:46:34,643 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15487.46 MB 2025-02-14 13:46:34,643 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16332.82 MB 2025-02-14 13:46:34,643 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 845.36 MB 2025-02-14 13:46:34,643 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17863.54 MB 2025-02-14 13:46:34,643 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18287.17 MB 2025-02-14 13:46:34,643 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 423.62 MB 2025-02-14 13:46:34,643 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16967.12 MB 2025-02-14 13:46:34,740 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 13:46:34,740 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 13:46:34,740 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 13:46:34,740 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:46:34,740 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16332.82 MB 2025-02-14 13:46:34,740 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17336.58 MB 2025-02-14 13:46:34,740 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1003.76 MB 2025-02-14 13:46:34,740 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18287.17 MB 2025-02-14 13:46:34,740 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20828.91 MB 2025-02-14 13:46:34,740 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2541.75 MB 2025-02-14 13:46:34,740 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19818.92 MB 2025-02-14 13:46:34,741 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 13:46:34,741 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 13:46:34,741 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 13:46:34,741 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:46:34,741 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15487.46 MB 2025-02-14 13:46:34,741 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17336.58 MB 2025-02-14 13:46:34,741 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1849.12 MB 2025-02-14 13:46:34,741 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17863.54 MB 2025-02-14 13:46:34,741 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20828.91 MB 2025-02-14 13:46:34,741 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2965.37 MB 2025-02-14 13:46:34,741 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19818.92 MB 2025-02-14 13:46:34,821 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 13:46:34,821 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 13:46:34,821 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 13:46:34,821 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:46:34,822 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18022.84 MB 2025-02-14 13:46:34,822 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18367.38 MB 2025-02-14 13:46:34,822 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 344.54 MB 2025-02-14 13:46:34,822 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20828.91 MB 2025-02-14 13:46:34,822 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21015.56 MB 2025-02-14 13:46:34,822 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 186.65 MB 2025-02-14 13:46:34,822 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18690.78 MB 2025-02-14 13:46:34,833 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 13:46:34,833 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 13:46:34,833 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:46:34,833 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:46:34,833 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18552.15 MB 2025-02-14 13:46:34,833 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18761.16 MB 2025-02-14 13:46:34,833 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 209.01 MB 2025-02-14 13:46:34,833 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21015.56 MB 2025-02-14 13:46:34,833 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21015.56 MB 2025-02-14 13:46:34,833 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:46:34,833 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18793.83 MB 2025-02-14 13:46:34,834 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 13:46:34,834 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 13:46:34,834 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.15 seconds 2025-02-14 13:46:34,834 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:46:34,834 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13655.07 MB 2025-02-14 13:46:34,834 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18961.80 MB 2025-02-14 13:46:34,834 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5306.72 MB 2025-02-14 13:46:34,834 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54540.63 MB 2025-02-14 13:46:34,834 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21015.56 MB 2025-02-14 13:46:34,834 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33525.07 MB 2025-02-14 13:46:34,834 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18961.80 MB 2025-02-14 13:46:35,101 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 13:46:35,101 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 13:46:35,101 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 13:46:35,101 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:46:35,101 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18961.80 MB 2025-02-14 13:46:35,101 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17610.82 MB 2025-02-14 13:46:35,101 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1350.98 MB 2025-02-14 13:46:35,101 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21015.56 MB 2025-02-14 13:46:35,101 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21015.56 MB 2025-02-14 13:46:35,101 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:46:35,101 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19062.04 MB 2025-02-14 13:46:35,119 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8144, cut from 8146 2025-02-14 13:46:35,119 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2.'] 2025-02-14 13:46:35,125 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 13:46:35,125 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 13:46:35,125 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:46:35,125 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:46:35,125 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17610.82 MB 2025-02-14 13:46:35,125 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26031.60 MB 2025-02-14 13:46:35,125 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8420.78 MB 2025-02-14 13:46:35,126 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21015.56 MB 2025-02-14 13:46:35,126 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31480.35 MB 2025-02-14 13:46:35,126 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10464.79 MB 2025-02-14 13:46:35,126 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26031.60 MB 2025-02-14 13:46:35,287 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7936] 2025-02-14 13:46:35,289 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:46:35,289 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 13:46:35,290 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:46:35,290 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 13:46:35,294 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 13:46:35,295 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:46:35,295 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 13:46:35,295 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2.'] 2025-02-14 13:47:28,360 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:47:28,360 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 13:47:28,365 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 13:47:28,368 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:47:28,368 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1270, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 13:47:28,369 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:47:28,369 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1270, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 13:47:47,830 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 13:47:47,830 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 13:47:47,830 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.45 seconds 2025-02-14 13:47:47,830 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:47:47,830 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21818.27 MB 2025-02-14 13:47:47,830 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26312.73 MB 2025-02-14 13:47:47,830 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4494.46 MB 2025-02-14 13:47:47,830 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39852.18 MB 2025-02-14 13:47:47,830 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38149.29 MB 2025-02-14 13:47:47,830 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1702.89 MB 2025-02-14 13:47:47,830 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35140.01 MB 2025-02-14 13:47:47,907 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 13:47:47,907 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 13:47:47,907 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 13:47:47,907 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:47:47,907 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26312.73 MB 2025-02-14 13:47:47,907 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22380.18 MB 2025-02-14 13:47:47,907 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3932.54 MB 2025-02-14 13:47:47,907 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38149.29 MB 2025-02-14 13:47:47,907 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46961.52 MB 2025-02-14 13:47:47,907 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8812.23 MB 2025-02-14 13:47:47,907 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39579.58 MB 2025-02-14 13:47:49,825 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 13:47:49,825 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 13:47:49,825 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 13:47:49,825 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:47:49,825 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22380.18 MB 2025-02-14 13:47:49,825 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22911.03 MB 2025-02-14 13:47:49,825 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 13:47:49,825 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46961.52 MB 2025-02-14 13:47:49,825 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33653.00 MB 2025-02-14 13:47:49,825 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13308.53 MB 2025-02-14 13:47:49,825 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26890.36 MB 2025-02-14 13:47:49,838 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 13:47:49,838 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 13:47:49,838 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:47:49,838 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:47:49,838 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22911.03 MB 2025-02-14 13:47:49,838 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24800.56 MB 2025-02-14 13:47:49,838 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 13:47:49,838 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33653.00 MB 2025-02-14 13:47:49,838 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33653.00 MB 2025-02-14 13:47:49,838 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:47:49,838 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26217.99 MB 2025-02-14 13:47:50,045 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 13:47:50,045 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 13:47:50,045 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 13:47:50,045 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:47:50,045 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24800.56 MB 2025-02-14 13:47:50,045 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27042.42 MB 2025-02-14 13:47:50,045 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 13:47:50,045 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33653.00 MB 2025-02-14 13:47:50,045 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35540.43 MB 2025-02-14 13:47:50,045 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 13:47:50,045 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32586.70 MB 2025-02-14 13:47:50,046 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 13:47:50,046 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 13:47:50,046 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 13:47:50,046 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:47:50,046 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22911.03 MB 2025-02-14 13:47:50,046 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27042.42 MB 2025-02-14 13:47:50,046 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 13:47:50,046 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33653.00 MB 2025-02-14 13:47:50,046 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35540.43 MB 2025-02-14 13:47:50,046 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 13:47:50,046 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32586.70 MB 2025-02-14 13:47:50,225 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 13:47:50,225 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 13:47:50,225 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 13:47:50,225 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:47:50,226 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28575.96 MB 2025-02-14 13:47:50,226 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29342.96 MB 2025-02-14 13:47:50,226 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 13:47:50,226 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35540.43 MB 2025-02-14 13:47:50,226 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35955.67 MB 2025-02-14 13:47:50,226 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 13:47:50,226 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30050.75 MB 2025-02-14 13:47:50,252 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 13:47:50,252 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 13:47:50,252 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:47:50,252 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:47:50,252 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29755.85 MB 2025-02-14 13:47:50,252 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29984.98 MB 2025-02-14 13:47:50,252 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.13 MB 2025-02-14 13:47:50,252 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35955.67 MB 2025-02-14 13:47:50,252 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35955.67 MB 2025-02-14 13:47:50,252 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:47:50,252 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30220.00 MB 2025-02-14 13:47:50,254 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 13:47:50,254 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 13:47:50,254 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.88 seconds 2025-02-14 13:47:50,254 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:47:50,254 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17393.49 MB 2025-02-14 13:47:50,254 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30186.03 MB 2025-02-14 13:47:50,254 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12792.54 MB 2025-02-14 13:47:50,254 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39852.18 MB 2025-02-14 13:47:50,254 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35955.67 MB 2025-02-14 13:47:50,254 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3896.51 MB 2025-02-14 13:47:50,254 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30220.00 MB 2025-02-14 13:47:50,539 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 13:47:50,539 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 13:47:50,540 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.28 seconds 2025-02-14 13:47:50,540 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:47:50,540 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30186.03 MB 2025-02-14 13:47:50,540 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22397.50 MB 2025-02-14 13:47:50,540 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7788.53 MB 2025-02-14 13:47:50,540 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35955.67 MB 2025-02-14 13:47:50,540 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35955.67 MB 2025-02-14 13:47:50,540 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:47:50,540 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32697.39 MB 2025-02-14 13:47:50,559 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8161, cut from 8163 2025-02-14 13:47:50,559 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 13:47:50,567 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 13:47:50,567 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 13:47:50,567 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:47:50,567 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:47:50,567 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22397.50 MB 2025-02-14 13:47:50,567 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30837.36 MB 2025-02-14 13:47:50,567 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.86 MB 2025-02-14 13:47:50,567 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35955.67 MB 2025-02-14 13:47:50,567 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44344.28 MB 2025-02-14 13:47:50,567 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8388.61 MB 2025-02-14 13:47:50,567 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30837.36 MB 2025-02-14 13:47:50,814 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7953] 2025-02-14 13:47:50,816 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:47:50,817 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 13:47:50,818 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:47:50,819 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 13:47:50,826 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 13:47:50,828 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:47:50,828 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 13:47:50,828 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 13:48:00,947 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:48:00,947 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 13:48:00,952 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 13:48:00,956 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:48:00,956 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1306, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 13:48:00,957 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:48:00,957 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1306, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 13:48:21,293 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 13:48:21,293 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 13:48:21,294 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.33 seconds 2025-02-14 13:48:21,294 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:48:21,294 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22069.12 MB 2025-02-14 13:48:21,294 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26691.25 MB 2025-02-14 13:48:21,294 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4622.12 MB 2025-02-14 13:48:21,294 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52732.89 MB 2025-02-14 13:48:21,294 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38273.02 MB 2025-02-14 13:48:21,294 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14459.86 MB 2025-02-14 13:48:21,294 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35617.36 MB 2025-02-14 13:48:21,370 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 13:48:21,370 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 13:48:21,370 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 13:48:21,370 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:48:21,371 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26691.25 MB 2025-02-14 13:48:21,371 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22567.34 MB 2025-02-14 13:48:21,371 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4123.91 MB 2025-02-14 13:48:21,371 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38273.02 MB 2025-02-14 13:48:21,371 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47445.97 MB 2025-02-14 13:48:21,371 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9172.94 MB 2025-02-14 13:48:21,371 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40521.48 MB 2025-02-14 13:48:23,303 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 13:48:23,303 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 13:48:23,303 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 13:48:23,303 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:48:23,303 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22567.34 MB 2025-02-14 13:48:23,303 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23098.18 MB 2025-02-14 13:48:23,303 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 13:48:23,303 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47445.97 MB 2025-02-14 13:48:23,303 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33650.90 MB 2025-02-14 13:48:23,303 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13795.07 MB 2025-02-14 13:48:23,303 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27077.51 MB 2025-02-14 13:48:23,316 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 13:48:23,316 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 13:48:23,316 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:48:23,316 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:48:23,316 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23098.18 MB 2025-02-14 13:48:23,316 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24987.71 MB 2025-02-14 13:48:23,316 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 13:48:23,316 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33650.90 MB 2025-02-14 13:48:23,316 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33650.90 MB 2025-02-14 13:48:23,316 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:48:23,316 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26405.14 MB 2025-02-14 13:48:23,530 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 13:48:23,530 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 13:48:23,530 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 13:48:23,530 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:48:23,530 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24987.71 MB 2025-02-14 13:48:23,530 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27229.57 MB 2025-02-14 13:48:23,530 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 13:48:23,530 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33650.90 MB 2025-02-14 13:48:23,530 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36010.20 MB 2025-02-14 13:48:23,530 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2359.30 MB 2025-02-14 13:48:23,530 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32773.85 MB 2025-02-14 13:48:23,531 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 13:48:23,531 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 13:48:23,531 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 13:48:23,531 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:48:23,531 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23098.18 MB 2025-02-14 13:48:23,531 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27229.57 MB 2025-02-14 13:48:23,531 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 13:48:23,531 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33650.90 MB 2025-02-14 13:48:23,531 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36010.20 MB 2025-02-14 13:48:23,531 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2359.30 MB 2025-02-14 13:48:23,531 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32773.85 MB 2025-02-14 13:48:23,698 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 13:48:23,698 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 13:48:23,698 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 13:48:23,698 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:48:23,698 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28763.11 MB 2025-02-14 13:48:23,698 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29530.11 MB 2025-02-14 13:48:23,698 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 13:48:23,698 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36010.20 MB 2025-02-14 13:48:23,698 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36427.53 MB 2025-02-14 13:48:23,698 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 13:48:23,698 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30237.90 MB 2025-02-14 13:48:23,717 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 13:48:23,717 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 13:48:23,717 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:48:23,717 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:48:23,717 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29943.00 MB 2025-02-14 13:48:23,717 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30171.89 MB 2025-02-14 13:48:23,717 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.89 MB 2025-02-14 13:48:23,717 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36427.53 MB 2025-02-14 13:48:23,717 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36427.53 MB 2025-02-14 13:48:23,717 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:48:23,717 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30404.06 MB 2025-02-14 13:48:23,718 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 13:48:23,718 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 13:48:23,718 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.76 seconds 2025-02-14 13:48:23,718 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:48:23,718 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17518.91 MB 2025-02-14 13:48:23,718 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30372.69 MB 2025-02-14 13:48:23,718 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12853.78 MB 2025-02-14 13:48:23,718 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52732.89 MB 2025-02-14 13:48:23,718 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36427.53 MB 2025-02-14 13:48:23,718 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16305.36 MB 2025-02-14 13:48:23,718 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30404.06 MB 2025-02-14 13:48:23,988 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 13:48:23,988 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 13:48:23,988 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 13:48:23,988 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:48:23,988 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30372.69 MB 2025-02-14 13:48:23,988 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22519.65 MB 2025-02-14 13:48:23,988 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7853.04 MB 2025-02-14 13:48:23,988 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36427.53 MB 2025-02-14 13:48:23,988 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36427.53 MB 2025-02-14 13:48:23,988 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:48:23,988 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32880.98 MB 2025-02-14 13:48:24,006 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8151, cut from 8153 2025-02-14 13:48:24,007 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 13:48:24,013 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 13:48:24,013 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 13:48:24,013 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:48:24,013 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:48:24,013 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22519.65 MB 2025-02-14 13:48:24,013 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30946.99 MB 2025-02-14 13:48:24,013 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8427.34 MB 2025-02-14 13:48:24,013 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36427.53 MB 2025-02-14 13:48:24,013 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44807.75 MB 2025-02-14 13:48:24,013 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8380.22 MB 2025-02-14 13:48:24,013 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30946.99 MB 2025-02-14 13:48:24,178 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7943] 2025-02-14 13:48:24,179 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:48:24,179 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 13:48:24,180 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:48:24,180 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 13:48:24,185 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 13:48:24,186 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:48:24,186 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 13:48:24,186 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 13:49:19,915 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:49:19,915 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 13:49:19,920 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 13:49:19,924 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:49:19,924 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 154, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 13:49:19,925 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:49:19,925 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 154, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 13:49:22,322 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 13:49:22,322 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 13:49:22,322 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.39 seconds 2025-02-14 13:49:22,322 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:49:22,322 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14041.80 MB 2025-02-14 13:49:22,322 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14586.80 MB 2025-02-14 13:49:22,322 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 545.00 MB 2025-02-14 13:49:22,322 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53187.97 MB 2025-02-14 13:49:22,322 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18324.91 MB 2025-02-14 13:49:22,322 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34863.05 MB 2025-02-14 13:49:22,322 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23513.98 MB 2025-02-14 13:49:22,334 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 13:49:22,334 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 13:49:22,334 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:49:22,334 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:49:22,334 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14586.80 MB 2025-02-14 13:49:22,334 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14851.11 MB 2025-02-14 13:49:22,334 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 264.31 MB 2025-02-14 13:49:22,334 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18324.91 MB 2025-02-14 13:49:22,334 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18870.17 MB 2025-02-14 13:49:22,334 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 545.26 MB 2025-02-14 13:49:22,334 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16785.61 MB 2025-02-14 13:49:23,078 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 13:49:23,078 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 13:49:23,078 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.74 seconds 2025-02-14 13:49:23,078 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:49:23,078 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14851.11 MB 2025-02-14 13:49:23,078 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15055.49 MB 2025-02-14 13:49:23,078 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 204.37 MB 2025-02-14 13:49:23,078 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18870.17 MB 2025-02-14 13:49:23,078 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18870.17 MB 2025-02-14 13:49:23,078 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:49:23,078 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19022.59 MB 2025-02-14 13:49:23,086 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 13:49:23,086 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 13:49:23,086 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 13:49:23,086 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:49:23,086 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15055.42 MB 2025-02-14 13:49:23,086 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15782.71 MB 2025-02-14 13:49:23,086 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 727.29 MB 2025-02-14 13:49:23,086 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18870.17 MB 2025-02-14 13:49:23,086 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18870.17 MB 2025-02-14 13:49:23,086 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:49:23,086 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16328.43 MB 2025-02-14 13:49:23,167 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 13:49:23,168 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 13:49:23,168 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 13:49:23,168 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:49:23,168 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15782.71 MB 2025-02-14 13:49:23,168 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16645.87 MB 2025-02-14 13:49:23,168 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 863.15 MB 2025-02-14 13:49:23,168 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18870.17 MB 2025-02-14 13:49:23,168 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19964.89 MB 2025-02-14 13:49:23,168 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1094.71 MB 2025-02-14 13:49:23,168 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18780.38 MB 2025-02-14 13:49:23,168 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 13:49:23,168 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 13:49:23,168 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 13:49:23,168 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:49:23,168 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15055.42 MB 2025-02-14 13:49:23,168 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16645.87 MB 2025-02-14 13:49:23,168 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1590.45 MB 2025-02-14 13:49:23,168 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18870.17 MB 2025-02-14 13:49:23,168 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19964.89 MB 2025-02-14 13:49:23,168 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1094.71 MB 2025-02-14 13:49:23,168 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18780.38 MB 2025-02-14 13:49:23,263 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 13:49:23,263 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 13:49:23,263 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 13:49:23,263 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:49:23,263 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17236.28 MB 2025-02-14 13:49:23,263 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17531.58 MB 2025-02-14 13:49:23,263 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 295.30 MB 2025-02-14 13:49:23,263 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19964.89 MB 2025-02-14 13:49:23,263 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20122.17 MB 2025-02-14 13:49:23,263 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 157.29 MB 2025-02-14 13:49:23,263 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17812.33 MB 2025-02-14 13:49:23,277 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 13:49:23,277 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 13:49:23,277 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:49:23,277 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:49:23,277 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17690.55 MB 2025-02-14 13:49:23,277 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17894.97 MB 2025-02-14 13:49:23,277 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 204.42 MB 2025-02-14 13:49:23,277 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20122.17 MB 2025-02-14 13:49:23,277 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20126.37 MB 2025-02-14 13:49:23,277 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-14 13:49:23,277 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17915.71 MB 2025-02-14 13:49:23,279 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 13:49:23,279 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 13:49:23,279 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.35 seconds 2025-02-14 13:49:23,279 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:49:23,279 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13505.25 MB 2025-02-14 13:49:23,279 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18095.67 MB 2025-02-14 13:49:23,279 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4590.42 MB 2025-02-14 13:49:23,279 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53187.97 MB 2025-02-14 13:49:23,279 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20126.37 MB 2025-02-14 13:49:23,279 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33061.60 MB 2025-02-14 13:49:23,279 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18095.67 MB 2025-02-14 13:49:23,563 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 13:49:23,563 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 13:49:23,563 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.28 seconds 2025-02-14 13:49:23,563 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:49:23,563 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18095.67 MB 2025-02-14 13:49:23,563 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17343.72 MB 2025-02-14 13:49:23,563 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -751.95 MB 2025-02-14 13:49:23,563 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20126.37 MB 2025-02-14 13:49:23,563 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20126.37 MB 2025-02-14 13:49:23,563 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:49:23,563 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18998.21 MB 2025-02-14 13:49:23,583 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8147, cut from 8149 2025-02-14 13:49:23,584 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2.'] 2025-02-14 13:49:23,591 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 13:49:23,591 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 13:49:23,591 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:49:23,591 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:49:23,591 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17343.72 MB 2025-02-14 13:49:23,591 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25766.93 MB 2025-02-14 13:49:23,591 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8423.21 MB 2025-02-14 13:49:23,591 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20126.37 MB 2025-02-14 13:49:23,591 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30597.45 MB 2025-02-14 13:49:23,591 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10471.08 MB 2025-02-14 13:49:23,591 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25766.93 MB 2025-02-14 13:49:23,824 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7939] 2025-02-14 13:49:23,827 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:49:23,827 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 13:49:23,829 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:49:23,829 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 13:49:23,836 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 13:49:23,838 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:49:23,838 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 13:49:23,838 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2.'] 2025-02-14 13:50:08,412 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:50:08,412 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 13:50:08,417 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 13:50:08,420 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:50:08,420 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1267, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 13:50:08,421 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:50:08,421 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1267, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 13:50:27,889 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 13:50:27,890 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 13:50:27,890 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.46 seconds 2025-02-14 13:50:27,890 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:50:27,890 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21797.37 MB 2025-02-14 13:50:27,890 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26281.21 MB 2025-02-14 13:50:27,890 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4483.84 MB 2025-02-14 13:50:27,890 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38973.47 MB 2025-02-14 13:50:27,890 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38143.00 MB 2025-02-14 13:50:27,890 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -830.47 MB 2025-02-14 13:50:27,890 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35119.11 MB 2025-02-14 13:50:27,965 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 13:50:27,965 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 13:50:27,965 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 13:50:27,965 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:50:27,965 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26281.21 MB 2025-02-14 13:50:27,965 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22364.59 MB 2025-02-14 13:50:27,965 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3916.62 MB 2025-02-14 13:50:27,965 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38143.00 MB 2025-02-14 13:50:27,965 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46965.72 MB 2025-02-14 13:50:27,965 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8822.72 MB 2025-02-14 13:50:27,965 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39560.15 MB 2025-02-14 13:50:29,890 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 13:50:29,890 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 13:50:29,890 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 13:50:29,890 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:50:29,890 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22364.59 MB 2025-02-14 13:50:29,890 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22895.43 MB 2025-02-14 13:50:29,890 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 13:50:29,890 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46965.72 MB 2025-02-14 13:50:29,890 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29469.18 MB 2025-02-14 13:50:29,890 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17496.54 MB 2025-02-14 13:50:29,890 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26874.76 MB 2025-02-14 13:50:29,903 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 13:50:29,903 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 13:50:29,903 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:50:29,903 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:50:29,903 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22895.43 MB 2025-02-14 13:50:29,904 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24784.96 MB 2025-02-14 13:50:29,904 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 13:50:29,904 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29469.18 MB 2025-02-14 13:50:29,904 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29471.28 MB 2025-02-14 13:50:29,904 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-14 13:50:29,904 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26202.39 MB 2025-02-14 13:50:30,117 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 13:50:30,118 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 13:50:30,118 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 13:50:30,118 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:50:30,118 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24784.96 MB 2025-02-14 13:50:30,118 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27026.82 MB 2025-02-14 13:50:30,118 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 13:50:30,118 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29471.28 MB 2025-02-14 13:50:30,118 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35133.59 MB 2025-02-14 13:50:30,118 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 13:50:30,118 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32571.10 MB 2025-02-14 13:50:30,118 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 13:50:30,118 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 13:50:30,118 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 13:50:30,118 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:50:30,118 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22895.43 MB 2025-02-14 13:50:30,118 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27026.82 MB 2025-02-14 13:50:30,118 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 13:50:30,118 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29469.18 MB 2025-02-14 13:50:30,118 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35133.59 MB 2025-02-14 13:50:30,118 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5664.41 MB 2025-02-14 13:50:30,118 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32571.10 MB 2025-02-14 13:50:30,288 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 13:50:30,288 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 13:50:30,288 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 13:50:30,288 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:50:30,288 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28560.36 MB 2025-02-14 13:50:30,288 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29327.36 MB 2025-02-14 13:50:30,288 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 13:50:30,288 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35133.59 MB 2025-02-14 13:50:30,288 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35550.92 MB 2025-02-14 13:50:30,288 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 13:50:30,288 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30035.15 MB 2025-02-14 13:50:30,306 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 13:50:30,307 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 13:50:30,307 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:50:30,307 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:50:30,307 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29740.25 MB 2025-02-14 13:50:30,307 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29968.23 MB 2025-02-14 13:50:30,307 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.98 MB 2025-02-14 13:50:30,307 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35550.92 MB 2025-02-14 13:50:30,307 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35550.92 MB 2025-02-14 13:50:30,307 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:50:30,307 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30192.21 MB 2025-02-14 13:50:30,308 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 13:50:30,308 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 13:50:30,308 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.88 seconds 2025-02-14 13:50:30,308 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:50:30,308 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17383.04 MB 2025-02-14 13:50:30,308 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30169.03 MB 2025-02-14 13:50:30,308 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12786.00 MB 2025-02-14 13:50:30,308 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38973.47 MB 2025-02-14 13:50:30,308 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35550.92 MB 2025-02-14 13:50:30,308 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3422.55 MB 2025-02-14 13:50:30,308 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30192.21 MB 2025-02-14 13:50:30,578 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 13:50:30,578 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 13:50:30,578 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 13:50:30,578 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:50:30,578 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30169.03 MB 2025-02-14 13:50:30,578 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22383.78 MB 2025-02-14 13:50:30,578 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7785.26 MB 2025-02-14 13:50:30,578 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35550.92 MB 2025-02-14 13:50:30,578 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35550.92 MB 2025-02-14 13:50:30,578 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:50:30,578 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32677.32 MB 2025-02-14 13:50:30,596 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8151, cut from 8153 2025-02-14 13:50:30,596 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 13:50:30,602 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 13:50:30,602 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 13:50:30,602 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:50:30,602 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:50:30,602 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22383.78 MB 2025-02-14 13:50:30,602 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30811.11 MB 2025-02-14 13:50:30,602 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8427.34 MB 2025-02-14 13:50:30,602 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35550.92 MB 2025-02-14 13:50:30,602 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43931.14 MB 2025-02-14 13:50:30,602 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8380.22 MB 2025-02-14 13:50:30,602 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30811.11 MB 2025-02-14 13:50:30,764 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7943] 2025-02-14 13:50:30,765 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:50:30,765 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 13:50:30,766 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:50:30,766 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 13:50:30,771 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 13:50:30,772 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:50:30,772 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 13:50:30,772 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 13:52:07,416 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:52:07,416 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 13:52:07,422 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 13:52:07,427 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:52:07,427 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1018, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 13:52:07,428 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:52:07,428 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1018, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 13:52:23,034 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 13:52:23,034 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 13:52:23,034 - resource_logging.py:150 - __exit__ - DEBUG - Time: 15.60 seconds 2025-02-14 13:52:23,034 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:52:23,034 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20062.29 MB 2025-02-14 13:52:23,034 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23665.20 MB 2025-02-14 13:52:23,034 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3602.91 MB 2025-02-14 13:52:23,034 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52311.36 MB 2025-02-14 13:52:23,034 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28886.17 MB 2025-02-14 13:52:23,034 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23425.19 MB 2025-02-14 13:52:23,034 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32478.87 MB 2025-02-14 13:52:23,100 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 13:52:23,100 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 13:52:23,100 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 13:52:23,100 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:52:23,100 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23665.20 MB 2025-02-14 13:52:23,100 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21070.11 MB 2025-02-14 13:52:23,100 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2595.09 MB 2025-02-14 13:52:23,100 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28886.17 MB 2025-02-14 13:52:23,100 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40470.84 MB 2025-02-14 13:52:23,100 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 11584.67 MB 2025-02-14 13:52:23,100 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34794.73 MB 2025-02-14 13:52:25,006 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 13:52:25,006 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 13:52:25,006 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.90 seconds 2025-02-14 13:52:25,006 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:52:25,006 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21070.11 MB 2025-02-14 13:52:25,006 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21600.96 MB 2025-02-14 13:52:25,006 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 13:52:25,006 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40470.84 MB 2025-02-14 13:52:25,006 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26698.84 MB 2025-02-14 13:52:25,006 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13772.00 MB 2025-02-14 13:52:25,006 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25580.29 MB 2025-02-14 13:52:25,020 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 13:52:25,020 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 13:52:25,020 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:52:25,020 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:52:25,020 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21600.96 MB 2025-02-14 13:52:25,020 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23490.49 MB 2025-02-14 13:52:25,020 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 13:52:25,020 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26698.84 MB 2025-02-14 13:52:25,020 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27642.56 MB 2025-02-14 13:52:25,020 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 13:52:25,020 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24907.92 MB 2025-02-14 13:52:25,230 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 13:52:25,230 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 13:52:25,230 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 13:52:25,230 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:52:25,231 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23490.49 MB 2025-02-14 13:52:25,231 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25732.35 MB 2025-02-14 13:52:25,231 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 13:52:25,231 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27642.56 MB 2025-02-14 13:52:25,231 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33776.73 MB 2025-02-14 13:52:25,231 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 13:52:25,231 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31276.63 MB 2025-02-14 13:52:25,231 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 13:52:25,231 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 13:52:25,231 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 13:52:25,231 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:52:25,231 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21600.96 MB 2025-02-14 13:52:25,231 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25732.35 MB 2025-02-14 13:52:25,231 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 13:52:25,231 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26698.84 MB 2025-02-14 13:52:25,231 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33776.73 MB 2025-02-14 13:52:25,231 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7077.89 MB 2025-02-14 13:52:25,231 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31276.63 MB 2025-02-14 13:52:25,401 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 13:52:25,401 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 13:52:25,401 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 13:52:25,401 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:52:25,401 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27265.89 MB 2025-02-14 13:52:25,401 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28032.89 MB 2025-02-14 13:52:25,401 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 13:52:25,401 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33776.73 MB 2025-02-14 13:52:25,401 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34194.06 MB 2025-02-14 13:52:25,401 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 13:52:25,401 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28740.68 MB 2025-02-14 13:52:25,420 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 13:52:25,420 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 13:52:25,420 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:52:25,420 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:52:25,420 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28445.78 MB 2025-02-14 13:52:25,420 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28674.93 MB 2025-02-14 13:52:25,420 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.15 MB 2025-02-14 13:52:25,420 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34194.06 MB 2025-02-14 13:52:25,420 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34194.06 MB 2025-02-14 13:52:25,420 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:52:25,420 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28877.10 MB 2025-02-14 13:52:25,421 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 13:52:25,421 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 13:52:25,421 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.99 seconds 2025-02-14 13:52:25,421 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:52:25,421 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16515.50 MB 2025-02-14 13:52:25,421 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28876.00 MB 2025-02-14 13:52:25,421 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12360.50 MB 2025-02-14 13:52:25,421 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52311.36 MB 2025-02-14 13:52:25,421 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34194.06 MB 2025-02-14 13:52:25,421 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18117.30 MB 2025-02-14 13:52:25,421 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28877.10 MB 2025-02-14 13:52:25,689 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 13:52:25,689 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 13:52:25,689 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 13:52:25,689 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:52:25,689 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28876.00 MB 2025-02-14 13:52:25,689 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21519.89 MB 2025-02-14 13:52:25,689 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7356.11 MB 2025-02-14 13:52:25,689 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34194.06 MB 2025-02-14 13:52:25,689 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34194.06 MB 2025-02-14 13:52:25,689 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:52:25,689 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31387.66 MB 2025-02-14 13:52:25,707 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 13:52:25,708 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 13:52:25,714 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 13:52:25,714 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 13:52:25,714 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:52:25,714 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:52:25,714 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21519.89 MB 2025-02-14 13:52:25,714 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29958.91 MB 2025-02-14 13:52:25,714 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 13:52:25,714 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34194.06 MB 2025-02-14 13:52:25,714 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42584.77 MB 2025-02-14 13:52:25,714 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 13:52:25,714 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29958.91 MB 2025-02-14 13:52:25,878 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 13:52:25,879 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:52:25,879 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 13:52:25,880 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:52:25,880 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 13:52:25,885 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 13:52:25,886 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:52:25,886 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 13:52:25,886 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 13:53:27,306 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:53:27,306 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 13:53:27,314 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 13:53:27,322 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:53:27,322 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1956, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 13:53:27,324 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:53:27,324 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1956, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 13:53:57,613 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 13:53:57,613 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 13:53:57,613 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.28 seconds 2025-02-14 13:53:57,613 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:53:57,613 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26598.43 MB 2025-02-14 13:53:57,613 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33521.13 MB 2025-02-14 13:53:57,613 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6922.70 MB 2025-02-14 13:53:57,613 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55169.78 MB 2025-02-14 13:53:57,613 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40596.67 MB 2025-02-14 13:53:57,613 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14573.11 MB 2025-02-14 13:53:57,613 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42412.39 MB 2025-02-14 13:53:57,737 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 13:53:57,737 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 13:53:57,737 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 13:53:57,737 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:53:57,737 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33521.13 MB 2025-02-14 13:53:57,737 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25946.48 MB 2025-02-14 13:53:57,737 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7574.64 MB 2025-02-14 13:53:57,737 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40596.67 MB 2025-02-14 13:53:57,737 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 63073.94 MB 2025-02-14 13:53:57,737 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 22477.28 MB 2025-02-14 13:53:57,737 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 53356.13 MB 2025-02-14 13:53:59,678 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 13:53:59,678 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 13:53:59,678 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-14 13:53:59,678 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:53:59,678 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25946.48 MB 2025-02-14 13:53:59,678 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26477.33 MB 2025-02-14 13:53:59,678 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 13:53:59,678 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63073.94 MB 2025-02-14 13:53:59,678 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35089.55 MB 2025-02-14 13:53:59,678 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27984.40 MB 2025-02-14 13:53:59,678 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30456.66 MB 2025-02-14 13:53:59,692 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 13:53:59,692 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 13:53:59,692 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:53:59,692 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:53:59,692 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26477.33 MB 2025-02-14 13:53:59,692 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28366.86 MB 2025-02-14 13:53:59,692 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 13:53:59,692 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35089.55 MB 2025-02-14 13:53:59,692 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35089.55 MB 2025-02-14 13:53:59,692 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:53:59,692 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29784.29 MB 2025-02-14 13:53:59,903 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 13:53:59,903 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 13:53:59,903 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 13:53:59,903 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:53:59,903 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28366.86 MB 2025-02-14 13:53:59,903 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30608.72 MB 2025-02-14 13:53:59,903 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 13:53:59,903 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35089.55 MB 2025-02-14 13:53:59,903 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39808.14 MB 2025-02-14 13:53:59,903 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 13:53:59,903 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36153.00 MB 2025-02-14 13:53:59,904 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 13:53:59,904 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 13:53:59,904 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 13:53:59,904 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:53:59,904 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26477.33 MB 2025-02-14 13:53:59,904 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30608.72 MB 2025-02-14 13:53:59,904 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 13:53:59,904 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35089.55 MB 2025-02-14 13:53:59,904 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39808.14 MB 2025-02-14 13:53:59,904 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 13:53:59,904 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36153.00 MB 2025-02-14 13:54:00,069 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 13:54:00,069 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 13:54:00,069 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 13:54:00,069 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:54:00,069 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32142.26 MB 2025-02-14 13:54:00,069 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32909.26 MB 2025-02-14 13:54:00,069 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 13:54:00,069 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39808.14 MB 2025-02-14 13:54:00,069 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40223.38 MB 2025-02-14 13:54:00,069 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 13:54:00,069 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33617.05 MB 2025-02-14 13:54:00,089 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 13:54:00,089 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 13:54:00,089 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:54:00,089 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:54:00,089 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33322.15 MB 2025-02-14 13:54:00,089 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33551.02 MB 2025-02-14 13:54:00,089 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.88 MB 2025-02-14 13:54:00,089 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40223.38 MB 2025-02-14 13:54:00,089 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40223.38 MB 2025-02-14 13:54:00,090 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:54:00,090 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33759.48 MB 2025-02-14 13:54:00,091 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 13:54:00,091 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 13:54:00,091 - resource_logging.py:150 - __exit__ - DEBUG - Time: 32.76 seconds 2025-02-14 13:54:00,091 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:54:00,091 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19783.57 MB 2025-02-14 13:54:00,091 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33752.10 MB 2025-02-14 13:54:00,091 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13968.53 MB 2025-02-14 13:54:00,091 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55169.78 MB 2025-02-14 13:54:00,091 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40223.38 MB 2025-02-14 13:54:00,091 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14946.40 MB 2025-02-14 13:54:00,091 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33759.48 MB 2025-02-14 13:54:00,362 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 13:54:00,362 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 13:54:00,362 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 13:54:00,362 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:54:00,362 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33752.10 MB 2025-02-14 13:54:00,362 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24787.96 MB 2025-02-14 13:54:00,362 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8964.14 MB 2025-02-14 13:54:00,362 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40223.38 MB 2025-02-14 13:54:00,362 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40223.38 MB 2025-02-14 13:54:00,362 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:54:00,362 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36263.76 MB 2025-02-14 13:54:00,380 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 13:54:00,380 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 13:54:00,386 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 13:54:00,386 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 13:54:00,386 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:54:00,386 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:54:00,386 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24787.96 MB 2025-02-14 13:54:00,386 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33226.98 MB 2025-02-14 13:54:00,386 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 13:54:00,386 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40223.38 MB 2025-02-14 13:54:00,386 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48614.08 MB 2025-02-14 13:54:00,386 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 13:54:00,386 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33226.98 MB 2025-02-14 13:54:00,547 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 13:54:00,548 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:54:00,548 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 13:54:00,549 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:54:00,549 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 13:54:00,554 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 13:54:00,555 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:54:00,555 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 13:54:00,555 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 13:54:45,774 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:54:45,774 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 13:54:45,779 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 13:54:45,783 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:54:45,783 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1502, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 13:54:45,784 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:54:45,784 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1502, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 13:55:09,031 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 13:55:09,031 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 13:55:09,031 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.24 seconds 2025-02-14 13:55:09,031 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:55:09,031 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23434.88 MB 2025-02-14 13:55:09,031 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28751.16 MB 2025-02-14 13:55:09,031 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5316.28 MB 2025-02-14 13:55:09,031 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61199.09 MB 2025-02-14 13:55:09,032 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38992.35 MB 2025-02-14 13:55:09,032 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22206.74 MB 2025-02-14 13:55:09,032 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37663.40 MB 2025-02-14 13:55:09,112 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 13:55:09,112 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 13:55:09,112 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 13:55:09,112 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:55:09,112 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28751.16 MB 2025-02-14 13:55:09,112 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23586.28 MB 2025-02-14 13:55:09,112 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5164.88 MB 2025-02-14 13:55:09,112 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38992.35 MB 2025-02-14 13:55:09,112 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48320.48 MB 2025-02-14 13:55:09,112 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9328.13 MB 2025-02-14 13:55:09,112 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42829.26 MB 2025-02-14 13:55:11,038 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 13:55:11,038 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 13:55:11,038 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 13:55:11,038 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:55:11,038 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23586.28 MB 2025-02-14 13:55:11,038 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24117.12 MB 2025-02-14 13:55:11,038 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 13:55:11,038 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48320.48 MB 2025-02-14 13:55:11,038 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33676.07 MB 2025-02-14 13:55:11,038 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14644.41 MB 2025-02-14 13:55:11,038 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28096.46 MB 2025-02-14 13:55:11,052 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 13:55:11,052 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 13:55:11,052 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:55:11,052 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:55:11,052 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24117.12 MB 2025-02-14 13:55:11,052 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26006.66 MB 2025-02-14 13:55:11,052 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 13:55:11,052 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33676.07 MB 2025-02-14 13:55:11,052 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33676.07 MB 2025-02-14 13:55:11,052 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:55:11,052 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27424.08 MB 2025-02-14 13:55:11,258 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 13:55:11,258 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 13:55:11,258 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 13:55:11,258 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:55:11,258 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26006.66 MB 2025-02-14 13:55:11,258 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28248.51 MB 2025-02-14 13:55:11,258 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 13:55:11,258 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33676.07 MB 2025-02-14 13:55:11,258 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37450.94 MB 2025-02-14 13:55:11,258 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 13:55:11,258 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33792.79 MB 2025-02-14 13:55:11,258 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 13:55:11,258 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 13:55:11,258 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 13:55:11,258 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:55:11,258 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24117.12 MB 2025-02-14 13:55:11,258 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28248.51 MB 2025-02-14 13:55:11,259 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 13:55:11,259 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33676.07 MB 2025-02-14 13:55:11,259 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37450.94 MB 2025-02-14 13:55:11,259 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 13:55:11,259 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33792.79 MB 2025-02-14 13:55:11,421 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 13:55:11,421 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 13:55:11,421 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 13:55:11,421 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:55:11,421 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29782.05 MB 2025-02-14 13:55:11,421 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30549.06 MB 2025-02-14 13:55:11,421 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 13:55:11,421 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37450.94 MB 2025-02-14 13:55:11,421 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37864.08 MB 2025-02-14 13:55:11,421 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 13:55:11,421 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31256.84 MB 2025-02-14 13:55:11,439 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 13:55:11,439 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 13:55:11,439 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:55:11,439 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:55:11,439 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30961.95 MB 2025-02-14 13:55:11,439 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31191.84 MB 2025-02-14 13:55:11,439 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.90 MB 2025-02-14 13:55:11,439 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37864.08 MB 2025-02-14 13:55:11,439 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37864.08 MB 2025-02-14 13:55:11,439 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:55:11,439 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31388.97 MB 2025-02-14 13:55:11,441 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 13:55:11,441 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 13:55:11,441 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.65 seconds 2025-02-14 13:55:11,441 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:55:11,441 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18201.79 MB 2025-02-14 13:55:11,441 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31392.91 MB 2025-02-14 13:55:11,441 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13191.12 MB 2025-02-14 13:55:11,441 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61199.09 MB 2025-02-14 13:55:11,441 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37864.08 MB 2025-02-14 13:55:11,441 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23335.01 MB 2025-02-14 13:55:11,441 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31392.91 MB 2025-02-14 13:55:11,709 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 13:55:11,709 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 13:55:11,709 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 13:55:11,709 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:55:11,709 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31392.91 MB 2025-02-14 13:55:11,709 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23206.18 MB 2025-02-14 13:55:11,709 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8186.73 MB 2025-02-14 13:55:11,709 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37864.08 MB 2025-02-14 13:55:11,709 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37864.08 MB 2025-02-14 13:55:11,709 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:55:11,709 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33904.58 MB 2025-02-14 13:55:11,727 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 13:55:11,728 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 13:55:11,734 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 13:55:11,734 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 13:55:11,734 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:55:11,734 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:55:11,734 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23206.18 MB 2025-02-14 13:55:11,734 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31645.21 MB 2025-02-14 13:55:11,734 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 13:55:11,734 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37864.08 MB 2025-02-14 13:55:11,734 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46254.78 MB 2025-02-14 13:55:11,734 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 13:55:11,734 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31645.21 MB 2025-02-14 13:55:11,885 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 13:55:11,886 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:55:11,887 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 13:55:11,887 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:55:11,887 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 13:55:11,892 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 13:55:11,893 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:55:11,893 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 13:55:11,893 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 13:55:21,252 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:55:21,252 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 13:55:21,257 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 13:55:21,260 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:55:21,260 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1161, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 13:55:21,261 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:55:21,261 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1161, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 13:55:39,503 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 13:55:39,504 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 13:55:39,504 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.24 seconds 2025-02-14 13:55:39,504 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:55:39,504 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21058.74 MB 2025-02-14 13:55:39,504 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25167.45 MB 2025-02-14 13:55:39,504 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4108.71 MB 2025-02-14 13:55:39,504 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58839.79 MB 2025-02-14 13:55:39,504 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29406.27 MB 2025-02-14 13:55:39,504 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29433.53 MB 2025-02-14 13:55:39,504 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34154.80 MB 2025-02-14 13:55:39,581 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 13:55:39,581 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 13:55:39,581 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 13:55:39,581 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:55:39,581 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25167.45 MB 2025-02-14 13:55:39,581 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21813.53 MB 2025-02-14 13:55:39,581 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3353.93 MB 2025-02-14 13:55:39,581 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29406.27 MB 2025-02-14 13:55:39,581 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44667.24 MB 2025-02-14 13:55:39,581 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 15260.98 MB 2025-02-14 13:55:39,581 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37578.67 MB 2025-02-14 13:55:41,504 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 13:55:41,504 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 13:55:41,504 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 13:55:41,504 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:55:41,504 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21813.53 MB 2025-02-14 13:55:41,504 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22344.37 MB 2025-02-14 13:55:41,504 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 13:55:41,504 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44667.24 MB 2025-02-14 13:55:41,504 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26711.43 MB 2025-02-14 13:55:41,504 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17955.82 MB 2025-02-14 13:55:41,504 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26323.70 MB 2025-02-14 13:55:41,518 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 13:55:41,518 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 13:55:41,518 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:55:41,518 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:55:41,518 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22344.37 MB 2025-02-14 13:55:41,518 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24233.90 MB 2025-02-14 13:55:41,518 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 13:55:41,518 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26711.43 MB 2025-02-14 13:55:41,518 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28598.86 MB 2025-02-14 13:55:41,518 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 13:55:41,518 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25651.33 MB 2025-02-14 13:55:41,728 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 13:55:41,728 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 13:55:41,728 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 13:55:41,728 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:55:41,728 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24233.90 MB 2025-02-14 13:55:41,728 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26475.76 MB 2025-02-14 13:55:41,728 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 13:55:41,728 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28598.86 MB 2025-02-14 13:55:41,728 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34261.17 MB 2025-02-14 13:55:41,728 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 13:55:41,728 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32020.04 MB 2025-02-14 13:55:41,729 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 13:55:41,729 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 13:55:41,729 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 13:55:41,729 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:55:41,729 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22344.37 MB 2025-02-14 13:55:41,729 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26475.76 MB 2025-02-14 13:55:41,729 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 13:55:41,729 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26711.43 MB 2025-02-14 13:55:41,729 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34261.17 MB 2025-02-14 13:55:41,729 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-14 13:55:41,729 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32020.04 MB 2025-02-14 13:55:41,897 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 13:55:41,897 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 13:55:41,897 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 13:55:41,897 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:55:41,897 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28009.30 MB 2025-02-14 13:55:41,897 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28776.30 MB 2025-02-14 13:55:41,897 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 13:55:41,897 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34261.17 MB 2025-02-14 13:55:41,897 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34678.51 MB 2025-02-14 13:55:41,897 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 13:55:41,897 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29484.09 MB 2025-02-14 13:55:41,917 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 13:55:41,917 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 13:55:41,917 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:55:41,917 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:55:41,917 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29189.19 MB 2025-02-14 13:55:41,917 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29416.47 MB 2025-02-14 13:55:41,917 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.28 MB 2025-02-14 13:55:41,917 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34678.51 MB 2025-02-14 13:55:41,917 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34678.51 MB 2025-02-14 13:55:41,917 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:55:41,917 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29643.57 MB 2025-02-14 13:55:41,918 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 13:55:41,918 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 13:55:41,918 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.65 seconds 2025-02-14 13:55:41,918 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:55:41,918 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17013.72 MB 2025-02-14 13:55:41,918 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29616.66 MB 2025-02-14 13:55:41,918 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12602.93 MB 2025-02-14 13:55:41,918 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58839.79 MB 2025-02-14 13:55:41,918 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34678.51 MB 2025-02-14 13:55:41,918 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24161.29 MB 2025-02-14 13:55:41,918 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29643.57 MB 2025-02-14 13:55:42,189 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 13:55:42,189 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 13:55:42,189 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 13:55:42,189 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:55:42,189 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29616.66 MB 2025-02-14 13:55:42,189 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22004.40 MB 2025-02-14 13:55:42,189 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7612.26 MB 2025-02-14 13:55:42,189 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34678.51 MB 2025-02-14 13:55:42,189 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34678.51 MB 2025-02-14 13:55:42,189 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:55:42,189 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32117.27 MB 2025-02-14 13:55:42,207 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8126, cut from 8128 2025-02-14 13:55:42,207 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 13:55:42,213 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 13:55:42,213 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 13:55:42,213 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:55:42,213 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:55:42,213 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22004.40 MB 2025-02-14 13:55:42,213 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30405.93 MB 2025-02-14 13:55:42,213 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8401.53 MB 2025-02-14 13:55:42,213 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34678.51 MB 2025-02-14 13:55:42,213 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43033.56 MB 2025-02-14 13:55:42,213 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8355.05 MB 2025-02-14 13:55:42,213 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30405.93 MB 2025-02-14 13:55:42,374 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7918] 2025-02-14 13:55:42,376 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:55:42,376 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 13:55:42,377 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:55:42,377 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 13:55:42,382 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 13:55:42,383 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:55:42,383 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 13:55:42,383 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 13:55:52,476 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:55:52,477 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 13:55:52,483 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 13:55:52,489 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:55:52,489 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 194, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 13:55:52,491 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:55:52,491 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 194, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 13:55:55,666 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 13:55:55,666 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 13:55:55,666 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.17 seconds 2025-02-14 13:55:55,666 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:55:55,666 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14320.53 MB 2025-02-14 13:55:55,666 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15007.08 MB 2025-02-14 13:55:55,667 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 686.56 MB 2025-02-14 13:55:55,667 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51388.61 MB 2025-02-14 13:55:55,667 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17853.05 MB 2025-02-14 13:55:55,667 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33535.56 MB 2025-02-14 13:55:55,667 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24019.20 MB 2025-02-14 13:55:55,689 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 13:55:55,689 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 13:55:55,689 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:55:55,689 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:55:55,689 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15007.08 MB 2025-02-14 13:55:55,689 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15333.29 MB 2025-02-14 13:55:55,689 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 326.20 MB 2025-02-14 13:55:55,689 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17853.05 MB 2025-02-14 13:55:55,689 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19220.40 MB 2025-02-14 13:55:55,689 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1367.34 MB 2025-02-14 13:55:55,689 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17726.36 MB 2025-02-14 13:55:56,671 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 13:55:56,671 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 13:55:56,671 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.98 seconds 2025-02-14 13:55:56,671 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:55:56,671 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15333.29 MB 2025-02-14 13:55:56,671 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15589.42 MB 2025-02-14 13:55:56,671 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 256.13 MB 2025-02-14 13:55:56,671 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19220.40 MB 2025-02-14 13:55:56,671 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18536.73 MB 2025-02-14 13:55:56,671 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -683.67 MB 2025-02-14 13:55:56,671 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19589.69 MB 2025-02-14 13:55:56,683 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 13:55:56,683 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 13:55:56,683 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:55:56,683 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:55:56,683 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15589.42 MB 2025-02-14 13:55:56,683 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16500.90 MB 2025-02-14 13:55:56,683 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 911.48 MB 2025-02-14 13:55:56,683 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18536.73 MB 2025-02-14 13:55:56,683 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18993.91 MB 2025-02-14 13:55:56,683 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 457.18 MB 2025-02-14 13:55:56,683 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17184.81 MB 2025-02-14 13:55:56,824 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 13:55:56,824 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 13:55:56,824 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-14 13:55:56,824 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:55:56,824 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16500.90 MB 2025-02-14 13:55:56,824 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17582.62 MB 2025-02-14 13:55:56,824 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1081.73 MB 2025-02-14 13:55:56,824 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18993.91 MB 2025-02-14 13:55:56,824 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21736.98 MB 2025-02-14 13:55:56,824 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2743.07 MB 2025-02-14 13:55:56,824 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20258.20 MB 2025-02-14 13:55:56,825 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 13:55:56,825 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 13:55:56,825 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 13:55:56,826 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:55:56,826 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15589.42 MB 2025-02-14 13:55:56,826 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17582.62 MB 2025-02-14 13:55:56,826 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1993.21 MB 2025-02-14 13:55:56,826 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18536.73 MB 2025-02-14 13:55:56,826 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21736.98 MB 2025-02-14 13:55:56,826 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3200.25 MB 2025-02-14 13:55:56,826 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20258.20 MB 2025-02-14 13:55:56,964 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 13:55:56,964 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 13:55:56,964 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 13:55:56,964 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:55:56,964 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18323.05 MB 2025-02-14 13:55:56,964 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18693.13 MB 2025-02-14 13:55:56,964 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 370.08 MB 2025-02-14 13:55:56,964 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21736.98 MB 2025-02-14 13:55:56,964 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21934.11 MB 2025-02-14 13:55:56,964 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 197.13 MB 2025-02-14 13:55:56,964 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19038.72 MB 2025-02-14 13:55:56,983 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 13:55:56,983 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 13:55:56,983 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:55:56,983 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:55:56,983 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18892.35 MB 2025-02-14 13:55:56,983 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19119.93 MB 2025-02-14 13:55:56,983 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.57 MB 2025-02-14 13:55:56,983 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21934.11 MB 2025-02-14 13:55:56,983 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21934.11 MB 2025-02-14 13:55:56,983 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:55:56,983 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19157.48 MB 2025-02-14 13:55:56,986 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 13:55:56,986 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 13:55:56,986 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.49 seconds 2025-02-14 13:55:56,986 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:55:56,986 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13644.62 MB 2025-02-14 13:55:56,986 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19321.00 MB 2025-02-14 13:55:56,986 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5676.38 MB 2025-02-14 13:55:56,986 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51388.61 MB 2025-02-14 13:55:56,986 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21936.21 MB 2025-02-14 13:55:56,986 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29452.40 MB 2025-02-14 13:55:56,986 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19321.00 MB 2025-02-14 13:55:57,279 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 13:55:57,279 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 13:55:57,279 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-14 13:55:57,279 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:55:57,279 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19321.00 MB 2025-02-14 13:55:57,279 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17672.18 MB 2025-02-14 13:55:57,280 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1648.82 MB 2025-02-14 13:55:57,280 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21936.21 MB 2025-02-14 13:55:57,280 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21936.21 MB 2025-02-14 13:55:57,280 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:55:57,280 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19321.00 MB 2025-02-14 13:55:57,299 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 13:55:57,300 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 13:55:57,307 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 13:55:57,307 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 13:55:57,307 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:55:57,307 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:55:57,307 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17672.18 MB 2025-02-14 13:55:57,307 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26111.20 MB 2025-02-14 13:55:57,307 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 13:55:57,307 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21936.21 MB 2025-02-14 13:55:57,307 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32426.16 MB 2025-02-14 13:55:57,307 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 13:55:57,307 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26111.20 MB 2025-02-14 13:55:57,569 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 13:55:57,572 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:55:57,572 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 13:55:57,574 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:55:57,574 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 13:55:57,582 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 13:55:57,584 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:55:57,584 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 13:55:57,584 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 13:56:40,763 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:56:40,763 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 13:56:40,768 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 13:56:40,772 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:56:40,772 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 181, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 13:56:40,773 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:56:40,773 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 181, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 13:56:43,570 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 13:56:43,570 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 13:56:43,570 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.79 seconds 2025-02-14 13:56:43,570 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:56:43,570 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14229.94 MB 2025-02-14 13:56:43,570 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14870.49 MB 2025-02-14 13:56:43,570 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 640.55 MB 2025-02-14 13:56:43,570 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45011.17 MB 2025-02-14 13:56:43,570 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17823.69 MB 2025-02-14 13:56:43,570 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27187.48 MB 2025-02-14 13:56:43,570 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23702.12 MB 2025-02-14 13:56:43,584 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 13:56:43,584 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 13:56:43,584 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:56:43,584 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:56:43,584 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14870.49 MB 2025-02-14 13:56:43,584 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15153.40 MB 2025-02-14 13:56:43,584 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 282.91 MB 2025-02-14 13:56:43,584 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17823.69 MB 2025-02-14 13:56:43,584 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18450.74 MB 2025-02-14 13:56:43,584 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 627.05 MB 2025-02-14 13:56:43,584 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17375.06 MB 2025-02-14 13:56:44,436 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 13:56:44,437 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 13:56:44,437 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.85 seconds 2025-02-14 13:56:44,437 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:56:44,437 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15153.40 MB 2025-02-14 13:56:44,437 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15388.30 MB 2025-02-14 13:56:44,437 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 234.90 MB 2025-02-14 13:56:44,437 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18450.74 MB 2025-02-14 13:56:44,437 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18450.74 MB 2025-02-14 13:56:44,437 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:56:44,437 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19324.87 MB 2025-02-14 13:56:44,445 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 13:56:44,445 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 13:56:44,445 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:56:44,445 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:56:44,445 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15388.23 MB 2025-02-14 13:56:44,445 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16224.15 MB 2025-02-14 13:56:44,445 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 835.92 MB 2025-02-14 13:56:44,445 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18450.74 MB 2025-02-14 13:56:44,445 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18450.74 MB 2025-02-14 13:56:44,445 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:56:44,445 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16851.36 MB 2025-02-14 13:56:44,541 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 13:56:44,541 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 13:56:44,541 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 13:56:44,541 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:56:44,541 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16224.15 MB 2025-02-14 13:56:44,541 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17216.21 MB 2025-02-14 13:56:44,541 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 992.06 MB 2025-02-14 13:56:44,541 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18450.74 MB 2025-02-14 13:56:44,541 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20967.33 MB 2025-02-14 13:56:44,541 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2516.58 MB 2025-02-14 13:56:44,541 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19669.51 MB 2025-02-14 13:56:44,542 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 13:56:44,542 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 13:56:44,542 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 13:56:44,542 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:56:44,542 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15388.23 MB 2025-02-14 13:56:44,542 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17216.21 MB 2025-02-14 13:56:44,542 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1827.97 MB 2025-02-14 13:56:44,542 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18450.74 MB 2025-02-14 13:56:44,542 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20967.33 MB 2025-02-14 13:56:44,542 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2516.58 MB 2025-02-14 13:56:44,542 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19669.51 MB 2025-02-14 13:56:44,618 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 13:56:44,618 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 13:56:44,618 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 13:56:44,618 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:56:44,618 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17894.80 MB 2025-02-14 13:56:44,618 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18234.20 MB 2025-02-14 13:56:44,618 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 339.40 MB 2025-02-14 13:56:44,618 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20967.33 MB 2025-02-14 13:56:44,618 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21147.68 MB 2025-02-14 13:56:44,618 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 180.36 MB 2025-02-14 13:56:44,618 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18554.77 MB 2025-02-14 13:56:44,628 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 13:56:44,628 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 13:56:44,628 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:56:44,628 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:56:44,628 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18416.91 MB 2025-02-14 13:56:44,628 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18646.14 MB 2025-02-14 13:56:44,628 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.23 MB 2025-02-14 13:56:44,628 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21147.68 MB 2025-02-14 13:56:44,628 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21147.68 MB 2025-02-14 13:56:44,628 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:56:44,629 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18688.00 MB 2025-02-14 13:56:44,630 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 13:56:44,630 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 13:56:44,630 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.85 seconds 2025-02-14 13:56:44,630 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:56:44,630 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13599.32 MB 2025-02-14 13:56:44,630 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18847.21 MB 2025-02-14 13:56:44,630 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5247.89 MB 2025-02-14 13:56:44,630 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45011.17 MB 2025-02-14 13:56:44,630 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21147.68 MB 2025-02-14 13:56:44,630 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23863.49 MB 2025-02-14 13:56:44,630 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18847.21 MB 2025-02-14 13:56:44,899 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 13:56:44,899 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 13:56:44,899 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 13:56:44,899 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:56:44,899 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18847.21 MB 2025-02-14 13:56:44,899 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17551.03 MB 2025-02-14 13:56:44,899 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1296.19 MB 2025-02-14 13:56:44,899 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21147.68 MB 2025-02-14 13:56:44,899 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21147.68 MB 2025-02-14 13:56:44,899 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:56:44,899 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19082.19 MB 2025-02-14 13:56:44,917 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 13:56:44,917 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2.'] 2025-02-14 13:56:44,923 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 13:56:44,923 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 13:56:44,923 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:56:44,923 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:56:44,923 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17551.03 MB 2025-02-14 13:56:44,923 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25990.05 MB 2025-02-14 13:56:44,923 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 13:56:44,923 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21147.68 MB 2025-02-14 13:56:44,923 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31637.64 MB 2025-02-14 13:56:44,923 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 13:56:44,923 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25990.05 MB 2025-02-14 13:56:45,086 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 13:56:45,087 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:56:45,087 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 13:56:45,088 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:56:45,088 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 13:56:45,093 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 13:56:45,094 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:56:45,094 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 13:56:45,094 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2.'] 2025-02-14 13:57:56,694 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:57:56,694 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 13:57:56,699 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 13:57:56,703 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:57:56,703 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 865, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 13:57:56,704 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:57:56,704 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 865, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 13:58:09,955 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 13:58:09,955 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 13:58:09,955 - resource_logging.py:150 - __exit__ - DEBUG - Time: 13.25 seconds 2025-02-14 13:58:09,955 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:58:09,955 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18996.16 MB 2025-02-14 13:58:09,955 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22058.01 MB 2025-02-14 13:58:09,955 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3061.84 MB 2025-02-14 13:58:09,955 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44222.64 MB 2025-02-14 13:58:09,955 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30459.04 MB 2025-02-14 13:58:09,955 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13763.61 MB 2025-02-14 13:58:09,955 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30958.95 MB 2025-02-14 13:58:10,008 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 13:58:10,008 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 13:58:10,008 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-14 13:58:10,008 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:58:10,008 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22058.01 MB 2025-02-14 13:58:10,008 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20274.72 MB 2025-02-14 13:58:10,008 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1783.29 MB 2025-02-14 13:58:10,008 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30459.04 MB 2025-02-14 13:58:10,008 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37612.42 MB 2025-02-14 13:58:10,008 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7153.39 MB 2025-02-14 13:58:10,008 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32059.26 MB 2025-02-14 13:58:11,912 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 13:58:11,912 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 13:58:11,912 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.90 seconds 2025-02-14 13:58:11,912 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:58:11,912 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20274.72 MB 2025-02-14 13:58:11,912 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20805.56 MB 2025-02-14 13:58:11,912 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 13:58:11,912 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37612.42 MB 2025-02-14 13:58:11,912 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28812.77 MB 2025-02-14 13:58:11,912 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8799.65 MB 2025-02-14 13:58:11,912 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24784.89 MB 2025-02-14 13:58:11,926 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 13:58:11,926 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 13:58:11,926 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:58:11,926 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:58:11,926 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20805.56 MB 2025-02-14 13:58:11,926 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22695.09 MB 2025-02-14 13:58:11,926 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 13:58:11,926 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28812.77 MB 2025-02-14 13:58:11,926 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28812.77 MB 2025-02-14 13:58:11,926 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:58:11,926 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24112.52 MB 2025-02-14 13:58:12,139 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 13:58:12,139 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 13:58:12,139 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 13:58:12,139 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:58:12,139 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22695.09 MB 2025-02-14 13:58:12,139 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24936.95 MB 2025-02-14 13:58:12,139 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 13:58:12,139 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28812.77 MB 2025-02-14 13:58:12,139 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33531.36 MB 2025-02-14 13:58:12,139 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 13:58:12,139 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30481.23 MB 2025-02-14 13:58:12,140 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 13:58:12,140 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 13:58:12,140 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 13:58:12,140 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:58:12,140 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20805.56 MB 2025-02-14 13:58:12,140 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24936.95 MB 2025-02-14 13:58:12,140 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 13:58:12,140 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28812.77 MB 2025-02-14 13:58:12,140 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33531.36 MB 2025-02-14 13:58:12,140 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 13:58:12,140 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30481.23 MB 2025-02-14 13:58:12,305 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 13:58:12,305 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 13:58:12,305 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 13:58:12,305 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:58:12,305 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26470.49 MB 2025-02-14 13:58:12,305 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27237.49 MB 2025-02-14 13:58:12,305 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 13:58:12,305 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33531.36 MB 2025-02-14 13:58:12,305 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33946.60 MB 2025-02-14 13:58:12,305 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 13:58:12,305 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27945.28 MB 2025-02-14 13:58:12,324 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 13:58:12,325 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 13:58:12,325 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:58:12,325 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:58:12,325 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27650.38 MB 2025-02-14 13:58:12,325 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27877.84 MB 2025-02-14 13:58:12,325 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.46 MB 2025-02-14 13:58:12,325 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33946.60 MB 2025-02-14 13:58:12,325 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33946.60 MB 2025-02-14 13:58:12,325 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:58:12,325 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28104.71 MB 2025-02-14 13:58:12,326 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 13:58:12,326 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 13:58:12,326 - resource_logging.py:150 - __exit__ - DEBUG - Time: 15.62 seconds 2025-02-14 13:58:12,326 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:58:12,326 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15982.44 MB 2025-02-14 13:58:12,326 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28078.32 MB 2025-02-14 13:58:12,326 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12095.89 MB 2025-02-14 13:58:12,326 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44222.64 MB 2025-02-14 13:58:12,326 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33946.60 MB 2025-02-14 13:58:12,326 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10276.04 MB 2025-02-14 13:58:12,326 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28104.71 MB 2025-02-14 13:58:12,593 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 13:58:12,593 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 13:58:12,593 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 13:58:12,593 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:58:12,593 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28078.32 MB 2025-02-14 13:58:12,593 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20977.68 MB 2025-02-14 13:58:12,593 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7100.64 MB 2025-02-14 13:58:12,593 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33946.60 MB 2025-02-14 13:58:12,593 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33946.60 MB 2025-02-14 13:58:12,593 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:58:12,593 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30582.62 MB 2025-02-14 13:58:12,611 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8138, cut from 8140 2025-02-14 13:58:12,611 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 13:58:12,618 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 13:58:12,618 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 13:58:12,618 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:58:12,618 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:58:12,618 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20977.68 MB 2025-02-14 13:58:12,618 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29391.66 MB 2025-02-14 13:58:12,618 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8413.98 MB 2025-02-14 13:58:12,618 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33946.60 MB 2025-02-14 13:58:12,618 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42312.14 MB 2025-02-14 13:58:12,618 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8365.54 MB 2025-02-14 13:58:12,618 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29391.66 MB 2025-02-14 13:58:12,768 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7930] 2025-02-14 13:58:12,770 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:58:12,770 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 13:58:12,770 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:58:12,771 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 13:58:12,775 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 13:58:12,776 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:58:12,776 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 13:58:12,776 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 13:58:22,443 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:58:22,443 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 13:58:22,448 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 13:58:22,452 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:58:22,452 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1814, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 13:58:22,453 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:58:22,453 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1814, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 13:58:50,666 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 13:58:50,666 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 13:58:50,666 - resource_logging.py:150 - __exit__ - DEBUG - Time: 28.21 seconds 2025-02-14 13:58:50,666 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:58:50,666 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25608.95 MB 2025-02-14 13:58:50,666 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32028.59 MB 2025-02-14 13:58:50,666 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6419.64 MB 2025-02-14 13:58:50,666 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54859.40 MB 2025-02-14 13:58:50,666 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40059.80 MB 2025-02-14 13:58:50,667 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14799.60 MB 2025-02-14 13:58:50,667 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40969.93 MB 2025-02-14 13:58:50,779 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 13:58:50,779 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 13:58:50,779 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 13:58:50,779 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:58:50,779 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32028.59 MB 2025-02-14 13:58:50,779 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25208.27 MB 2025-02-14 13:58:50,779 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6820.32 MB 2025-02-14 13:58:50,779 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40059.80 MB 2025-02-14 13:58:50,779 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 59919.83 MB 2025-02-14 13:58:50,779 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 19860.03 MB 2025-02-14 13:58:50,779 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50673.93 MB 2025-02-14 13:58:52,723 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 13:58:52,723 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 13:58:52,723 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-14 13:58:52,723 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:58:52,723 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25208.27 MB 2025-02-14 13:58:52,723 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25739.11 MB 2025-02-14 13:58:52,723 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 13:58:52,723 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59919.83 MB 2025-02-14 13:58:52,723 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30872.17 MB 2025-02-14 13:58:52,723 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29047.65 MB 2025-02-14 13:58:52,723 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29718.45 MB 2025-02-14 13:58:52,739 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 13:58:52,740 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 13:58:52,740 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:58:52,740 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:58:52,740 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25739.11 MB 2025-02-14 13:58:52,740 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27628.65 MB 2025-02-14 13:58:52,740 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 13:58:52,740 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30872.17 MB 2025-02-14 13:58:52,740 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31815.89 MB 2025-02-14 13:58:52,740 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 13:58:52,740 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29046.08 MB 2025-02-14 13:58:52,952 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 13:58:52,952 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 13:58:52,952 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 13:58:52,952 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:58:52,952 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27628.65 MB 2025-02-14 13:58:52,952 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29870.50 MB 2025-02-14 13:58:52,952 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 13:58:52,952 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31815.89 MB 2025-02-14 13:58:52,952 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37950.06 MB 2025-02-14 13:58:52,952 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 13:58:52,952 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35414.78 MB 2025-02-14 13:58:52,953 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 13:58:52,953 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 13:58:52,953 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 13:58:52,953 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:58:52,953 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25739.11 MB 2025-02-14 13:58:52,953 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29870.50 MB 2025-02-14 13:58:52,953 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 13:58:52,953 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30872.17 MB 2025-02-14 13:58:52,953 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37950.06 MB 2025-02-14 13:58:52,953 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7077.89 MB 2025-02-14 13:58:52,953 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35414.78 MB 2025-02-14 13:58:53,124 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 13:58:53,124 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 13:58:53,124 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 13:58:53,124 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:58:53,124 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31404.04 MB 2025-02-14 13:58:53,124 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32171.05 MB 2025-02-14 13:58:53,124 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 13:58:53,124 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37950.06 MB 2025-02-14 13:58:53,124 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38365.30 MB 2025-02-14 13:58:53,124 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 13:58:53,124 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32878.84 MB 2025-02-14 13:58:53,144 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 13:58:53,144 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 13:58:53,144 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:58:53,144 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:58:53,144 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32583.94 MB 2025-02-14 13:58:53,144 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32812.13 MB 2025-02-14 13:58:53,144 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.20 MB 2025-02-14 13:58:53,144 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38365.30 MB 2025-02-14 13:58:53,144 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38365.30 MB 2025-02-14 13:58:53,144 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:58:53,144 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33016.77 MB 2025-02-14 13:58:53,145 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 13:58:53,145 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 13:58:53,145 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.69 seconds 2025-02-14 13:58:53,145 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:58:53,145 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19288.83 MB 2025-02-14 13:58:53,145 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33012.49 MB 2025-02-14 13:58:53,145 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13723.67 MB 2025-02-14 13:58:53,145 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54859.40 MB 2025-02-14 13:58:53,145 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38365.30 MB 2025-02-14 13:58:53,145 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16494.10 MB 2025-02-14 13:58:53,145 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33016.77 MB 2025-02-14 13:58:53,419 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 13:58:53,419 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 13:58:53,419 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 13:58:53,419 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:58:53,419 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33012.49 MB 2025-02-14 13:58:53,419 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24282.17 MB 2025-02-14 13:58:53,419 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8730.32 MB 2025-02-14 13:58:53,419 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38365.30 MB 2025-02-14 13:58:53,419 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38365.30 MB 2025-02-14 13:58:53,419 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:58:53,419 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35515.25 MB 2025-02-14 13:58:53,437 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8133, cut from 8135 2025-02-14 13:58:53,438 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 13:58:53,444 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 13:58:53,444 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 13:58:53,444 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:58:53,444 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:58:53,444 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24282.17 MB 2025-02-14 13:58:53,444 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32691.47 MB 2025-02-14 13:58:53,444 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8409.30 MB 2025-02-14 13:58:53,444 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38365.30 MB 2025-02-14 13:58:53,444 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46724.55 MB 2025-02-14 13:58:53,444 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8359.25 MB 2025-02-14 13:58:53,444 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32691.47 MB 2025-02-14 13:58:53,606 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7925] 2025-02-14 13:58:53,608 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:58:53,608 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 13:58:53,609 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:58:53,609 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 13:58:53,613 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 13:58:53,614 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:58:53,614 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 13:58:53,615 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 13:59:10,393 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:59:10,393 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 13:59:10,400 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 13:59:10,406 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:59:10,406 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 146, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 13:59:10,408 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:59:10,408 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 146, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 13:59:12,788 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 13:59:12,789 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 13:59:12,789 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.37 seconds 2025-02-14 13:59:12,789 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:59:12,789 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13986.06 MB 2025-02-14 13:59:12,789 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14502.74 MB 2025-02-14 13:59:12,789 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 516.69 MB 2025-02-14 13:59:12,789 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55083.79 MB 2025-02-14 13:59:12,789 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17855.15 MB 2025-02-14 13:59:12,789 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -37228.64 MB 2025-02-14 13:59:12,789 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23458.24 MB 2025-02-14 13:59:12,800 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 13:59:12,800 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 13:59:12,800 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:59:12,801 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:59:12,801 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14502.74 MB 2025-02-14 13:59:12,801 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 13910.31 MB 2025-02-14 13:59:12,801 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -592.43 MB 2025-02-14 13:59:12,801 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17855.15 MB 2025-02-14 13:59:12,801 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17855.15 MB 2025-02-14 13:59:12,801 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:59:12,801 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14877.02 MB 2025-02-14 13:59:12,945 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 13:59:12,945 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 13:59:12,946 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-14 13:59:12,946 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:59:12,946 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13910.31 MB 2025-02-14 13:59:12,946 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 13944.82 MB 2025-02-14 13:59:12,946 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 34.50 MB 2025-02-14 13:59:12,946 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17855.15 MB 2025-02-14 13:59:12,946 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17383.29 MB 2025-02-14 13:59:12,946 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -471.86 MB 2025-02-14 13:59:12,946 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15569.73 MB 2025-02-14 13:59:12,953 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 13:59:12,953 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 13:59:12,953 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 13:59:12,954 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:59:12,954 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13944.75 MB 2025-02-14 13:59:12,954 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14067.54 MB 2025-02-14 13:59:12,954 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 122.79 MB 2025-02-14 13:59:12,954 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17383.29 MB 2025-02-14 13:59:12,954 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17383.29 MB 2025-02-14 13:59:12,954 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:59:12,954 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14159.68 MB 2025-02-14 13:59:12,981 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 13:59:12,981 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 13:59:12,981 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:59:12,981 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:59:12,981 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14067.54 MB 2025-02-14 13:59:12,982 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14213.32 MB 2025-02-14 13:59:12,982 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 145.78 MB 2025-02-14 13:59:12,982 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17383.29 MB 2025-02-14 13:59:12,982 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17383.29 MB 2025-02-14 13:59:12,982 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:59:12,982 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14573.64 MB 2025-02-14 13:59:12,983 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 13:59:12,983 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 13:59:12,983 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 13:59:12,983 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:59:12,983 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13944.75 MB 2025-02-14 13:59:12,983 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14213.32 MB 2025-02-14 13:59:12,983 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 268.57 MB 2025-02-14 13:59:12,983 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17383.29 MB 2025-02-14 13:59:12,983 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17383.29 MB 2025-02-14 13:59:12,983 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:59:12,983 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14573.64 MB 2025-02-14 13:59:13,007 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 13:59:13,007 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 13:59:13,007 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 13:59:13,007 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:59:13,007 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14313.00 MB 2025-02-14 13:59:13,007 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14362.86 MB 2025-02-14 13:59:13,007 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 49.86 MB 2025-02-14 13:59:13,007 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17383.29 MB 2025-02-14 13:59:13,007 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17404.26 MB 2025-02-14 13:59:13,007 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 20.97 MB 2025-02-14 13:59:13,007 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14428.75 MB 2025-02-14 13:59:13,012 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 13:59:13,013 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 13:59:13,013 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 13:59:13,013 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:59:13,013 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14389.71 MB 2025-02-14 13:59:13,013 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14421.39 MB 2025-02-14 13:59:13,013 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 31.68 MB 2025-02-14 13:59:13,013 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17404.26 MB 2025-02-14 13:59:13,013 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17404.26 MB 2025-02-14 13:59:13,013 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 13:59:13,013 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14421.39 MB 2025-02-14 13:59:13,015 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 13:59:13,015 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 13:59:13,015 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.60 seconds 2025-02-14 13:59:13,015 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:59:13,015 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13477.38 MB 2025-02-14 13:59:13,015 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14480.92 MB 2025-02-14 13:59:13,015 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1003.54 MB 2025-02-14 13:59:13,015 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55083.79 MB 2025-02-14 13:59:13,015 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17404.26 MB 2025-02-14 13:59:13,015 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -37679.53 MB 2025-02-14 13:59:13,015 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14480.92 MB 2025-02-14 13:59:13,132 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 13:59:13,132 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 13:59:13,132 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 13:59:13,132 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:59:13,132 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14480.92 MB 2025-02-14 13:59:13,133 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15373.41 MB 2025-02-14 13:59:13,133 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 892.49 MB 2025-02-14 13:59:13,133 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17404.26 MB 2025-02-14 13:59:13,133 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17406.36 MB 2025-02-14 13:59:13,133 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-14 13:59:13,133 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15462.65 MB 2025-02-14 13:59:13,141 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 2407, cut from 2409 2025-02-14 13:59:13,142 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['2 final rate for this video is 2 ('] 2025-02-14 13:59:13,145 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 13:59:13,145 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 13:59:13,145 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 13:59:13,145 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 13:59:13,145 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14524.24 MB 2025-02-14 13:59:13,145 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17022.66 MB 2025-02-14 13:59:13,145 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2498.43 MB 2025-02-14 13:59:13,145 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17406.36 MB 2025-02-14 13:59:13,145 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18649.97 MB 2025-02-14 13:59:13,145 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1243.61 MB 2025-02-14 13:59:13,145 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17022.66 MB 2025-02-14 13:59:13,223 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 2199] 2025-02-14 13:59:13,225 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:59:13,225 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 13:59:13,227 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:59:13,227 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 13:59:13,235 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 13:59:13,237 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 13:59:13,237 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 13:59:13,237 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['2 final rate for this video is 2 ('] 2025-02-14 14:01:33,439 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:01:33,440 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 14:01:33,445 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 14:01:33,449 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:01:33,449 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 379, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 14:01:33,450 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:01:33,450 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 379, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 14:01:39,271 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 14:01:39,272 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 14:01:39,272 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.82 seconds 2025-02-14 14:01:39,272 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:01:39,272 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15609.64 MB 2025-02-14 14:01:39,272 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16951.82 MB 2025-02-14 14:01:39,272 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1342.18 MB 2025-02-14 14:01:39,272 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21137.20 MB 2025-02-14 14:01:39,272 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20738.74 MB 2025-02-14 14:01:39,272 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -398.46 MB 2025-02-14 14:01:39,272 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25761.29 MB 2025-02-14 14:01:39,301 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 14:01:39,301 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 14:01:39,301 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 14:01:39,301 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:01:39,301 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16951.82 MB 2025-02-14 14:01:39,301 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17602.33 MB 2025-02-14 14:01:39,301 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 650.51 MB 2025-02-14 14:01:39,301 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20738.74 MB 2025-02-14 14:01:39,301 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25394.41 MB 2025-02-14 14:01:39,301 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4655.68 MB 2025-02-14 14:01:39,301 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22317.05 MB 2025-02-14 14:01:41,103 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 14:01:41,103 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 14:01:41,103 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.80 seconds 2025-02-14 14:01:41,103 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:01:41,103 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17602.33 MB 2025-02-14 14:01:41,103 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18105.30 MB 2025-02-14 14:01:41,104 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 502.97 MB 2025-02-14 14:01:41,104 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25394.41 MB 2025-02-14 14:01:41,104 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22059.94 MB 2025-02-14 14:01:41,104 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3334.47 MB 2025-02-14 14:01:41,104 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22028.61 MB 2025-02-14 14:01:41,117 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 14:01:41,117 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 14:01:41,117 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 14:01:41,117 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:01:41,117 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18105.30 MB 2025-02-14 14:01:41,117 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19895.74 MB 2025-02-14 14:01:41,117 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1790.44 MB 2025-02-14 14:01:41,117 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22059.94 MB 2025-02-14 14:01:41,117 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23850.91 MB 2025-02-14 14:01:41,117 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1790.97 MB 2025-02-14 14:01:41,117 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21238.76 MB 2025-02-14 14:01:41,321 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 14:01:41,321 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 14:01:41,321 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 14:01:41,321 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:01:41,321 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19895.74 MB 2025-02-14 14:01:41,321 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22019.91 MB 2025-02-14 14:01:41,321 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2124.16 MB 2025-02-14 14:01:41,321 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23850.91 MB 2025-02-14 14:01:41,321 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29672.60 MB 2025-02-14 14:01:41,321 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5821.69 MB 2025-02-14 14:01:41,321 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27273.11 MB 2025-02-14 14:01:41,322 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 14:01:41,322 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 14:01:41,322 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 14:01:41,322 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:01:41,322 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18105.30 MB 2025-02-14 14:01:41,322 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22019.91 MB 2025-02-14 14:01:41,322 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3914.61 MB 2025-02-14 14:01:41,322 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22059.94 MB 2025-02-14 14:01:41,322 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29672.60 MB 2025-02-14 14:01:41,322 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7612.66 MB 2025-02-14 14:01:41,322 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27273.11 MB 2025-02-14 14:01:41,485 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 14:01:41,485 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 14:01:41,485 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 14:01:41,485 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:01:41,485 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23472.94 MB 2025-02-14 14:01:41,485 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24199.67 MB 2025-02-14 14:01:41,485 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 726.73 MB 2025-02-14 14:01:41,485 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29672.60 MB 2025-02-14 14:01:41,485 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30064.77 MB 2025-02-14 14:01:41,485 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 392.17 MB 2025-02-14 14:01:41,485 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24870.30 MB 2025-02-14 14:01:41,505 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 14:01:41,505 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 14:01:41,505 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:01:41,505 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:01:41,505 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24590.88 MB 2025-02-14 14:01:41,505 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24802.01 MB 2025-02-14 14:01:41,505 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 211.13 MB 2025-02-14 14:01:41,505 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30064.77 MB 2025-02-14 14:01:41,505 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30066.87 MB 2025-02-14 14:01:41,505 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-14 14:01:41,505 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24975.24 MB 2025-02-14 14:01:41,507 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 14:01:41,507 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 14:01:41,507 - resource_logging.py:150 - __exit__ - DEBUG - Time: 8.05 seconds 2025-02-14 14:01:41,507 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:01:41,507 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14289.17 MB 2025-02-14 14:01:41,507 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25003.08 MB 2025-02-14 14:01:41,507 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 10713.91 MB 2025-02-14 14:01:41,507 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21137.20 MB 2025-02-14 14:01:41,507 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30066.87 MB 2025-02-14 14:01:41,507 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8929.67 MB 2025-02-14 14:01:41,507 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25003.08 MB 2025-02-14 14:01:41,776 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 14:01:41,776 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 14:01:41,777 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 14:01:41,777 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:01:41,777 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25003.08 MB 2025-02-14 14:01:41,777 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19193.93 MB 2025-02-14 14:01:41,777 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5809.15 MB 2025-02-14 14:01:41,777 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30066.87 MB 2025-02-14 14:01:41,777 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30066.87 MB 2025-02-14 14:01:41,777 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:01:41,777 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27816.15 MB 2025-02-14 14:01:41,795 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 14:01:41,795 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-14 14:01:41,801 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 14:01:41,801 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 14:01:41,801 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:01:41,801 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:01:41,801 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19193.93 MB 2025-02-14 14:01:41,801 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27632.96 MB 2025-02-14 14:01:41,801 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 14:01:41,801 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30066.87 MB 2025-02-14 14:01:41,801 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40556.82 MB 2025-02-14 14:01:41,801 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 14:01:41,801 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27632.96 MB 2025-02-14 14:01:41,964 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 14:01:41,966 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:01:41,966 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 14:01:41,967 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:01:41,967 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 14:01:41,972 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 14:01:41,973 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:01:41,973 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 14:01:41,973 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-14 14:02:40,465 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:02:40,465 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 14:02:40,473 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 14:02:40,480 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:02:40,480 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 3112, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 14:02:40,482 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:02:40,482 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 3112, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 14:03:28,675 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 14:03:28,675 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 14:03:28,675 - resource_logging.py:150 - __exit__ - DEBUG - Time: 48.18 seconds 2025-02-14 14:03:28,675 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:03:28,675 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34654.67 MB 2025-02-14 14:03:28,675 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45668.85 MB 2025-02-14 14:03:28,675 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11014.18 MB 2025-02-14 14:03:28,675 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 74830.58 MB 2025-02-14 14:03:28,675 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49612.32 MB 2025-02-14 14:03:28,675 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25218.25 MB 2025-02-14 14:03:28,675 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 56682.04 MB 2025-02-14 14:03:28,893 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 14:03:28,893 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 14:03:28,893 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 14:03:28,893 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:03:28,893 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45668.85 MB 2025-02-14 14:03:28,893 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31956.69 MB 2025-02-14 14:03:28,893 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -13712.15 MB 2025-02-14 14:03:28,893 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49612.32 MB 2025-02-14 14:03:28,893 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 91330.97 MB 2025-02-14 14:03:28,893 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 41718.64 MB 2025-02-14 14:03:28,893 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 77422.37 MB 2025-02-14 14:03:30,873 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 14:03:30,873 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 14:03:30,873 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.98 seconds 2025-02-14 14:03:30,873 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:03:30,873 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31956.69 MB 2025-02-14 14:03:30,873 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32487.53 MB 2025-02-14 14:03:30,873 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 14:03:30,873 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 91330.97 MB 2025-02-14 14:03:30,873 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34506.54 MB 2025-02-14 14:03:30,873 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -56824.43 MB 2025-02-14 14:03:30,873 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36467.91 MB 2025-02-14 14:03:30,888 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 14:03:30,888 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 14:03:30,888 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 14:03:30,888 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:03:30,888 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32487.53 MB 2025-02-14 14:03:30,888 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34377.07 MB 2025-02-14 14:03:30,888 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 14:03:30,888 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34506.54 MB 2025-02-14 14:03:30,888 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37809.55 MB 2025-02-14 14:03:30,888 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 14:03:30,888 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35794.50 MB 2025-02-14 14:03:31,101 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 14:03:31,101 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 14:03:31,101 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 14:03:31,101 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:03:31,101 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34377.07 MB 2025-02-14 14:03:31,101 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36618.92 MB 2025-02-14 14:03:31,101 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 14:03:31,101 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37809.55 MB 2025-02-14 14:03:31,101 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44415.58 MB 2025-02-14 14:03:31,101 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 14:03:31,101 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42163.21 MB 2025-02-14 14:03:31,102 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 14:03:31,102 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 14:03:31,102 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 14:03:31,102 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:03:31,102 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32487.53 MB 2025-02-14 14:03:31,102 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36618.92 MB 2025-02-14 14:03:31,102 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 14:03:31,102 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34506.54 MB 2025-02-14 14:03:31,102 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44415.58 MB 2025-02-14 14:03:31,102 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9909.04 MB 2025-02-14 14:03:31,102 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42163.21 MB 2025-02-14 14:03:31,273 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 14:03:31,273 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 14:03:31,274 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 14:03:31,274 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:03:31,274 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38152.47 MB 2025-02-14 14:03:31,274 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38919.47 MB 2025-02-14 14:03:31,274 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 14:03:31,274 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44415.58 MB 2025-02-14 14:03:31,274 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44830.82 MB 2025-02-14 14:03:31,274 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 14:03:31,274 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39627.26 MB 2025-02-14 14:03:31,293 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 14:03:31,293 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 14:03:31,293 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:03:31,293 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:03:31,293 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39332.36 MB 2025-02-14 14:03:31,293 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39561.92 MB 2025-02-14 14:03:31,293 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.56 MB 2025-02-14 14:03:31,293 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44830.82 MB 2025-02-14 14:03:31,293 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44830.82 MB 2025-02-14 14:03:31,293 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:03:31,293 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39789.59 MB 2025-02-14 14:03:31,295 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 14:03:31,295 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 14:03:31,295 - resource_logging.py:150 - __exit__ - DEBUG - Time: 50.81 seconds 2025-02-14 14:03:31,295 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:03:31,295 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23811.69 MB 2025-02-14 14:03:31,295 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39762.99 MB 2025-02-14 14:03:31,295 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 15951.31 MB 2025-02-14 14:03:31,295 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 63986.20 MB 2025-02-14 14:03:31,295 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44830.82 MB 2025-02-14 14:03:31,295 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19155.39 MB 2025-02-14 14:03:31,295 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39789.59 MB 2025-02-14 14:03:31,564 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 14:03:31,564 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 14:03:31,564 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 14:03:31,564 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:03:31,564 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25802.04 MB 2025-02-14 14:03:31,564 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28816.08 MB 2025-02-14 14:03:31,564 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-14 14:03:31,564 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44830.82 MB 2025-02-14 14:03:31,564 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44830.82 MB 2025-02-14 14:03:31,564 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:03:31,564 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29117.44 MB 2025-02-14 14:03:31,582 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 14:03:31,583 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 14:03:31,589 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 14:03:31,589 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 14:03:31,589 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:03:31,589 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:03:31,589 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28816.08 MB 2025-02-14 14:03:31,589 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37254.77 MB 2025-02-14 14:03:31,589 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8438.69 MB 2025-02-14 14:03:31,589 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44830.82 MB 2025-02-14 14:03:31,589 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49027.22 MB 2025-02-14 14:03:31,589 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4196.40 MB 2025-02-14 14:03:31,589 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37254.77 MB 2025-02-14 14:03:31,754 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 14:03:31,756 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:03:31,756 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 14:03:31,757 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:03:31,757 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 14:03:31,762 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 14:03:31,764 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:03:31,764 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 14:03:31,764 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 14:04:29,242 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:04:29,243 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 14:04:29,247 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 14:04:29,251 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:04:29,251 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1445, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 14:04:29,252 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:04:29,252 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1445, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 14:04:51,662 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 14:04:51,662 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 14:04:51,662 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.40 seconds 2025-02-14 14:04:51,662 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:04:51,662 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23037.70 MB 2025-02-14 14:04:51,662 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28151.47 MB 2025-02-14 14:04:51,662 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5113.77 MB 2025-02-14 14:04:51,662 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57415.83 MB 2025-02-14 14:04:51,662 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39948.65 MB 2025-02-14 14:04:51,662 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17467.18 MB 2025-02-14 14:04:51,662 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37038.92 MB 2025-02-14 14:04:51,747 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 14:04:51,747 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 14:04:51,747 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 14:04:51,747 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:04:51,747 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28151.47 MB 2025-02-14 14:04:51,747 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23289.96 MB 2025-02-14 14:04:51,747 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4861.52 MB 2025-02-14 14:04:51,747 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39948.65 MB 2025-02-14 14:04:51,747 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49572.48 MB 2025-02-14 14:04:51,747 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9623.83 MB 2025-02-14 14:04:51,747 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42641.09 MB 2025-02-14 14:04:53,677 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 14:04:53,677 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 14:04:53,677 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 14:04:53,677 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:04:53,677 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23289.96 MB 2025-02-14 14:04:53,677 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23820.80 MB 2025-02-14 14:04:53,677 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 14:04:53,677 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49572.48 MB 2025-02-14 14:04:53,677 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30637.29 MB 2025-02-14 14:04:53,677 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18935.19 MB 2025-02-14 14:04:53,677 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27800.13 MB 2025-02-14 14:04:53,690 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 14:04:53,690 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 14:04:53,690 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 14:04:53,690 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:04:53,690 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23820.80 MB 2025-02-14 14:04:53,690 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25710.33 MB 2025-02-14 14:04:53,690 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 14:04:53,690 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30637.29 MB 2025-02-14 14:04:53,690 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30637.29 MB 2025-02-14 14:04:53,690 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:04:53,690 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27127.76 MB 2025-02-14 14:04:53,906 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 14:04:53,906 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 14:04:53,906 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 14:04:53,906 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:04:53,906 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25710.33 MB 2025-02-14 14:04:53,906 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27952.19 MB 2025-02-14 14:04:53,906 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 14:04:53,906 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30637.29 MB 2025-02-14 14:04:53,906 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35827.74 MB 2025-02-14 14:04:53,906 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 14:04:53,906 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33496.47 MB 2025-02-14 14:04:53,907 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 14:04:53,907 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 14:04:53,907 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 14:04:53,907 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:04:53,907 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23820.80 MB 2025-02-14 14:04:53,907 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27952.19 MB 2025-02-14 14:04:53,907 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 14:04:53,907 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30637.29 MB 2025-02-14 14:04:53,907 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35827.74 MB 2025-02-14 14:04:53,907 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 14:04:53,907 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33496.47 MB 2025-02-14 14:04:54,080 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 14:04:54,081 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 14:04:54,081 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 14:04:54,081 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:04:54,081 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29485.73 MB 2025-02-14 14:04:54,081 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30252.73 MB 2025-02-14 14:04:54,081 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 14:04:54,081 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35827.74 MB 2025-02-14 14:04:54,081 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36240.88 MB 2025-02-14 14:04:54,081 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 14:04:54,081 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30960.52 MB 2025-02-14 14:04:54,100 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 14:04:54,100 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 14:04:54,100 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:04:54,100 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:04:54,100 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30665.62 MB 2025-02-14 14:04:54,100 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30894.26 MB 2025-02-14 14:04:54,100 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.64 MB 2025-02-14 14:04:54,100 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36240.88 MB 2025-02-14 14:04:54,100 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36240.88 MB 2025-02-14 14:04:54,100 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:04:54,100 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31103.84 MB 2025-02-14 14:04:54,101 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 14:04:54,101 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 14:04:54,101 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.85 seconds 2025-02-14 14:04:54,101 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:04:54,101 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18003.20 MB 2025-02-14 14:04:54,101 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31095.33 MB 2025-02-14 14:04:54,101 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13092.13 MB 2025-02-14 14:04:54,101 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57415.83 MB 2025-02-14 14:04:54,101 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36240.88 MB 2025-02-14 14:04:54,101 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21174.94 MB 2025-02-14 14:04:54,101 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31103.84 MB 2025-02-14 14:04:54,374 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 14:04:54,374 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 14:04:54,374 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 14:04:54,374 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:04:54,374 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31095.33 MB 2025-02-14 14:04:54,374 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23007.59 MB 2025-02-14 14:04:54,374 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8087.74 MB 2025-02-14 14:04:54,374 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36240.88 MB 2025-02-14 14:04:54,374 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36240.88 MB 2025-02-14 14:04:54,374 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:04:54,374 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33607.00 MB 2025-02-14 14:04:54,392 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 14:04:54,392 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 14:04:54,398 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 14:04:54,398 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 14:04:54,398 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:04:54,398 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:04:54,398 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23007.59 MB 2025-02-14 14:04:54,398 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31446.28 MB 2025-02-14 14:04:54,398 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8438.69 MB 2025-02-14 14:04:54,398 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36240.88 MB 2025-02-14 14:04:54,398 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40437.28 MB 2025-02-14 14:04:54,398 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4196.40 MB 2025-02-14 14:04:54,398 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31446.28 MB 2025-02-14 14:04:54,561 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 14:04:54,563 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:04:54,563 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 14:04:54,564 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:04:54,564 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 14:04:54,569 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 14:04:54,570 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:04:54,570 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 14:04:54,570 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 14:05:38,569 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:05:38,569 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 14:05:38,574 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 14:05:38,578 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:05:38,578 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1215, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 14:05:38,579 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:05:38,579 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1215, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 14:05:57,365 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 14:05:57,365 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 14:05:57,365 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.78 seconds 2025-02-14 14:05:57,365 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:05:57,365 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21435.02 MB 2025-02-14 14:05:57,365 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25734.84 MB 2025-02-14 14:05:57,365 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4299.82 MB 2025-02-14 14:05:57,365 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48825.89 MB 2025-02-14 14:05:57,365 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33793.51 MB 2025-02-14 14:05:57,365 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15032.39 MB 2025-02-14 14:05:57,365 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34613.00 MB 2025-02-14 14:05:57,439 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 14:05:57,439 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 14:05:57,439 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 14:05:57,439 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:05:57,439 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25734.84 MB 2025-02-14 14:05:57,439 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22094.26 MB 2025-02-14 14:05:57,440 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3640.58 MB 2025-02-14 14:05:57,440 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33793.51 MB 2025-02-14 14:05:57,440 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43102.77 MB 2025-02-14 14:05:57,440 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9309.26 MB 2025-02-14 14:05:57,440 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38620.75 MB 2025-02-14 14:05:59,369 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 14:05:59,369 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 14:05:59,369 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 14:05:59,369 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:05:59,369 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22094.26 MB 2025-02-14 14:05:59,369 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22625.10 MB 2025-02-14 14:05:59,369 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 14:05:59,369 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43102.77 MB 2025-02-14 14:05:59,369 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29492.25 MB 2025-02-14 14:05:59,369 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13610.52 MB 2025-02-14 14:05:59,369 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26604.43 MB 2025-02-14 14:05:59,382 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 14:05:59,382 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 14:05:59,382 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 14:05:59,382 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:05:59,382 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22625.10 MB 2025-02-14 14:05:59,383 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24514.63 MB 2025-02-14 14:05:59,383 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 14:05:59,383 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29492.25 MB 2025-02-14 14:05:59,383 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29492.25 MB 2025-02-14 14:05:59,383 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:05:59,383 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25932.06 MB 2025-02-14 14:05:59,596 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 14:05:59,596 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 14:05:59,596 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 14:05:59,596 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:05:59,596 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24514.63 MB 2025-02-14 14:05:59,596 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26756.49 MB 2025-02-14 14:05:59,596 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 14:05:59,596 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29492.25 MB 2025-02-14 14:05:59,596 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34682.70 MB 2025-02-14 14:05:59,596 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 14:05:59,596 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32300.77 MB 2025-02-14 14:05:59,597 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 14:05:59,597 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 14:05:59,597 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 14:05:59,597 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:05:59,597 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22625.10 MB 2025-02-14 14:05:59,597 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26756.49 MB 2025-02-14 14:05:59,597 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 14:05:59,597 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29492.25 MB 2025-02-14 14:05:59,597 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34682.70 MB 2025-02-14 14:05:59,597 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 14:05:59,597 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32300.77 MB 2025-02-14 14:05:59,767 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 14:05:59,767 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 14:05:59,767 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 14:05:59,767 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:05:59,767 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28290.03 MB 2025-02-14 14:05:59,767 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29057.03 MB 2025-02-14 14:05:59,767 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 14:05:59,767 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34682.70 MB 2025-02-14 14:05:59,767 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35097.94 MB 2025-02-14 14:05:59,767 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 14:05:59,767 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29764.82 MB 2025-02-14 14:05:59,787 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 14:05:59,787 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 14:05:59,787 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:05:59,787 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:05:59,787 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29469.92 MB 2025-02-14 14:05:59,787 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29697.67 MB 2025-02-14 14:05:59,787 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.74 MB 2025-02-14 14:05:59,787 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35097.94 MB 2025-02-14 14:05:59,787 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35097.94 MB 2025-02-14 14:05:59,787 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:05:59,787 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29936.37 MB 2025-02-14 14:05:59,788 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 14:05:59,788 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 14:05:59,788 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.21 seconds 2025-02-14 14:05:59,788 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:05:59,788 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17201.86 MB 2025-02-14 14:05:59,788 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29898.69 MB 2025-02-14 14:05:59,788 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12696.83 MB 2025-02-14 14:05:59,788 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48825.89 MB 2025-02-14 14:05:59,788 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35097.94 MB 2025-02-14 14:05:59,788 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13727.96 MB 2025-02-14 14:05:59,788 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29936.37 MB 2025-02-14 14:06:00,057 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 14:06:00,057 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 14:06:00,057 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 14:06:00,057 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:06:00,057 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29898.69 MB 2025-02-14 14:06:00,057 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22205.59 MB 2025-02-14 14:06:00,057 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7693.10 MB 2025-02-14 14:06:00,057 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35097.94 MB 2025-02-14 14:06:00,057 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35097.94 MB 2025-02-14 14:06:00,057 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:06:00,057 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32409.74 MB 2025-02-14 14:06:00,075 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8160, cut from 8162 2025-02-14 14:06:00,075 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 14:06:00,081 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 14:06:00,081 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 14:06:00,081 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:06:00,081 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:06:00,081 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22205.59 MB 2025-02-14 14:06:00,081 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30643.06 MB 2025-02-14 14:06:00,081 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8437.47 MB 2025-02-14 14:06:00,081 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35097.94 MB 2025-02-14 14:06:00,081 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43486.54 MB 2025-02-14 14:06:00,081 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8388.61 MB 2025-02-14 14:06:00,081 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30643.06 MB 2025-02-14 14:06:00,247 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7952] 2025-02-14 14:06:00,249 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:06:00,249 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 14:06:00,250 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:06:00,250 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 14:06:00,254 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 14:06:00,255 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:06:00,255 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 14:06:00,256 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 14:07:04,043 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:07:04,043 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 14:07:04,048 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 14:07:04,052 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:07:04,052 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1002, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 14:07:04,053 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:07:04,054 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1002, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 14:07:19,511 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 14:07:19,512 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 14:07:19,512 - resource_logging.py:150 - __exit__ - DEBUG - Time: 15.45 seconds 2025-02-14 14:07:19,512 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:07:19,512 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19950.80 MB 2025-02-14 14:07:19,512 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23497.09 MB 2025-02-14 14:07:19,512 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3546.28 MB 2025-02-14 14:07:19,512 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51875.15 MB 2025-02-14 14:07:19,512 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28842.13 MB 2025-02-14 14:07:19,512 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23033.02 MB 2025-02-14 14:07:19,512 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32367.38 MB 2025-02-14 14:07:19,577 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 14:07:19,577 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 14:07:19,577 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 14:07:19,577 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:07:19,577 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23497.09 MB 2025-02-14 14:07:19,577 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20986.94 MB 2025-02-14 14:07:19,577 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2510.15 MB 2025-02-14 14:07:19,577 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28842.13 MB 2025-02-14 14:07:19,577 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40617.64 MB 2025-02-14 14:07:19,577 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 11775.51 MB 2025-02-14 14:07:19,577 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34766.34 MB 2025-02-14 14:07:21,489 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 14:07:21,489 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 14:07:21,489 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 14:07:21,489 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:07:21,489 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20986.94 MB 2025-02-14 14:07:21,489 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21517.78 MB 2025-02-14 14:07:21,489 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 14:07:21,489 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40617.64 MB 2025-02-14 14:07:21,489 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26711.43 MB 2025-02-14 14:07:21,489 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13906.21 MB 2025-02-14 14:07:21,489 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25497.11 MB 2025-02-14 14:07:21,503 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 14:07:21,503 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 14:07:21,503 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 14:07:21,503 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:07:21,503 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21517.78 MB 2025-02-14 14:07:21,503 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23407.31 MB 2025-02-14 14:07:21,503 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 14:07:21,503 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26711.43 MB 2025-02-14 14:07:21,503 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28598.86 MB 2025-02-14 14:07:21,503 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 14:07:21,503 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24824.74 MB 2025-02-14 14:07:21,709 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 14:07:21,709 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 14:07:21,709 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 14:07:21,709 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:07:21,709 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23407.31 MB 2025-02-14 14:07:21,709 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25649.17 MB 2025-02-14 14:07:21,709 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 14:07:21,709 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28598.86 MB 2025-02-14 14:07:21,709 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34261.17 MB 2025-02-14 14:07:21,709 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 14:07:21,709 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31193.45 MB 2025-02-14 14:07:21,710 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 14:07:21,710 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 14:07:21,710 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 14:07:21,710 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:07:21,710 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21517.78 MB 2025-02-14 14:07:21,710 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25649.17 MB 2025-02-14 14:07:21,710 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 14:07:21,710 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26711.43 MB 2025-02-14 14:07:21,710 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34261.17 MB 2025-02-14 14:07:21,710 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-14 14:07:21,710 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31193.45 MB 2025-02-14 14:07:21,873 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 14:07:21,873 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 14:07:21,873 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 14:07:21,873 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:07:21,873 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27182.71 MB 2025-02-14 14:07:21,873 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27949.71 MB 2025-02-14 14:07:21,873 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 14:07:21,873 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34261.17 MB 2025-02-14 14:07:21,873 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34678.51 MB 2025-02-14 14:07:21,873 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 14:07:21,873 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28657.50 MB 2025-02-14 14:07:21,892 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 14:07:21,892 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 14:07:21,892 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:07:21,892 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:07:21,892 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28362.60 MB 2025-02-14 14:07:21,892 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28590.40 MB 2025-02-14 14:07:21,892 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.80 MB 2025-02-14 14:07:21,892 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34678.51 MB 2025-02-14 14:07:21,892 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34678.51 MB 2025-02-14 14:07:21,892 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:07:21,892 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28813.53 MB 2025-02-14 14:07:21,893 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 14:07:21,893 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 14:07:21,893 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.84 seconds 2025-02-14 14:07:21,893 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:07:21,893 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16459.75 MB 2025-02-14 14:07:21,893 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28790.93 MB 2025-02-14 14:07:21,893 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12331.18 MB 2025-02-14 14:07:21,893 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51875.15 MB 2025-02-14 14:07:21,893 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34678.51 MB 2025-02-14 14:07:21,893 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17196.65 MB 2025-02-14 14:07:21,894 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28813.53 MB 2025-02-14 14:07:22,179 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 14:07:22,179 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 14:07:22,179 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.28 seconds 2025-02-14 14:07:22,179 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:07:22,179 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28790.93 MB 2025-02-14 14:07:22,179 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21455.76 MB 2025-02-14 14:07:22,179 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7335.17 MB 2025-02-14 14:07:22,179 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34678.51 MB 2025-02-14 14:07:22,179 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34678.51 MB 2025-02-14 14:07:22,179 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:07:22,179 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31295.84 MB 2025-02-14 14:07:22,199 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8140, cut from 8142 2025-02-14 14:07:22,199 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 14:07:22,206 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 14:07:22,207 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 14:07:22,207 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:07:22,207 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:07:22,207 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21455.76 MB 2025-02-14 14:07:22,207 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29872.37 MB 2025-02-14 14:07:22,207 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8416.60 MB 2025-02-14 14:07:22,207 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34678.51 MB 2025-02-14 14:07:22,207 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45139.10 MB 2025-02-14 14:07:22,207 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10460.59 MB 2025-02-14 14:07:22,207 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29872.37 MB 2025-02-14 14:07:22,476 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7932] 2025-02-14 14:07:22,479 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:07:22,479 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 14:07:22,481 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:07:22,481 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 14:07:22,489 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 14:07:22,491 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:07:22,491 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 14:07:22,491 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 14:09:48,666 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:09:48,667 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 14:09:48,672 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 14:09:48,677 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:09:48,677 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1538, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 14:09:48,679 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:09:48,679 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1538, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 14:10:12,259 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 14:10:12,259 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 14:10:12,259 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.57 seconds 2025-02-14 14:10:12,259 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:10:12,259 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23685.74 MB 2025-02-14 14:10:12,259 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29128.63 MB 2025-02-14 14:10:12,259 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5442.90 MB 2025-02-14 14:10:12,259 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53506.74 MB 2025-02-14 14:10:12,259 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39090.91 MB 2025-02-14 14:10:12,259 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14415.82 MB 2025-02-14 14:10:12,259 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38140.75 MB 2025-02-14 14:10:12,337 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 14:10:12,337 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 14:10:12,337 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 14:10:12,337 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:10:12,337 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29128.63 MB 2025-02-14 14:10:12,337 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23773.43 MB 2025-02-14 14:10:12,337 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5355.20 MB 2025-02-14 14:10:12,337 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39090.91 MB 2025-02-14 14:10:12,337 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46749.71 MB 2025-02-14 14:10:12,337 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7658.80 MB 2025-02-14 14:10:12,337 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41016.25 MB 2025-02-14 14:10:14,251 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 14:10:14,251 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 14:10:14,251 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 14:10:14,251 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:10:14,251 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23773.43 MB 2025-02-14 14:10:14,251 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24304.27 MB 2025-02-14 14:10:14,251 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 14:10:14,251 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46749.71 MB 2025-02-14 14:10:14,251 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33646.71 MB 2025-02-14 14:10:14,251 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13103.01 MB 2025-02-14 14:10:14,251 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28283.61 MB 2025-02-14 14:10:14,266 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 14:10:14,266 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 14:10:14,266 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 14:10:14,266 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:10:14,266 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24304.27 MB 2025-02-14 14:10:14,266 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26193.81 MB 2025-02-14 14:10:14,266 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 14:10:14,266 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33646.71 MB 2025-02-14 14:10:14,266 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33646.71 MB 2025-02-14 14:10:14,266 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:10:14,266 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27611.24 MB 2025-02-14 14:10:14,481 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 14:10:14,481 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 14:10:14,481 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 14:10:14,481 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:10:14,481 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26193.81 MB 2025-02-14 14:10:14,481 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28435.66 MB 2025-02-14 14:10:14,481 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 14:10:14,481 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33646.71 MB 2025-02-14 14:10:14,481 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37421.58 MB 2025-02-14 14:10:14,481 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 14:10:14,481 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33979.95 MB 2025-02-14 14:10:14,482 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 14:10:14,482 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 14:10:14,482 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 14:10:14,482 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:10:14,482 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24304.27 MB 2025-02-14 14:10:14,482 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28435.66 MB 2025-02-14 14:10:14,482 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 14:10:14,482 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33646.71 MB 2025-02-14 14:10:14,482 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37421.58 MB 2025-02-14 14:10:14,482 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 14:10:14,482 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33979.95 MB 2025-02-14 14:10:14,654 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 14:10:14,654 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 14:10:14,654 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 14:10:14,654 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:10:14,654 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29969.21 MB 2025-02-14 14:10:14,654 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30736.21 MB 2025-02-14 14:10:14,654 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 14:10:14,654 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37421.58 MB 2025-02-14 14:10:14,654 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37834.72 MB 2025-02-14 14:10:14,654 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 14:10:14,654 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31444.00 MB 2025-02-14 14:10:14,673 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 14:10:14,673 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 14:10:14,673 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:10:14,673 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:10:14,673 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31149.10 MB 2025-02-14 14:10:14,673 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31377.17 MB 2025-02-14 14:10:14,673 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.08 MB 2025-02-14 14:10:14,673 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37834.72 MB 2025-02-14 14:10:14,673 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37834.72 MB 2025-02-14 14:10:14,673 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:10:14,673 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31617.11 MB 2025-02-14 14:10:14,674 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 14:10:14,674 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 14:10:14,674 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.99 seconds 2025-02-14 14:10:14,674 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:10:14,674 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18327.22 MB 2025-02-14 14:10:14,674 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31577.16 MB 2025-02-14 14:10:14,674 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13249.94 MB 2025-02-14 14:10:14,674 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53506.74 MB 2025-02-14 14:10:14,674 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37834.72 MB 2025-02-14 14:10:14,674 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15672.02 MB 2025-02-14 14:10:14,674 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31617.11 MB 2025-02-14 14:10:14,942 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 14:10:14,942 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 14:10:14,942 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 14:10:14,942 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:10:14,942 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31577.16 MB 2025-02-14 14:10:14,943 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23314.85 MB 2025-02-14 14:10:14,943 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8262.32 MB 2025-02-14 14:10:14,943 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37834.72 MB 2025-02-14 14:10:14,943 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37834.72 MB 2025-02-14 14:10:14,943 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:10:14,943 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34075.32 MB 2025-02-14 14:10:14,960 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8118, cut from 8120 2025-02-14 14:10:14,960 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 14:10:14,966 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 14:10:14,966 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 14:10:14,966 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:10:14,966 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:10:14,966 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23314.85 MB 2025-02-14 14:10:14,966 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31708.12 MB 2025-02-14 14:10:14,966 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8393.27 MB 2025-02-14 14:10:14,966 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37834.72 MB 2025-02-14 14:10:14,966 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42008.05 MB 2025-02-14 14:10:14,966 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4173.33 MB 2025-02-14 14:10:14,966 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31708.12 MB 2025-02-14 14:10:15,131 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7910] 2025-02-14 14:10:15,133 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:10:15,133 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 14:10:15,134 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:10:15,134 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 14:10:15,138 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 14:10:15,139 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:10:15,139 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 14:10:15,140 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 14:10:46,210 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:10:46,210 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 14:10:46,218 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 14:10:46,223 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:10:46,224 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 3537, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 14:10:46,225 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:10:46,225 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 3537, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 14:11:41,565 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 14:11:41,566 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 14:11:41,566 - resource_logging.py:150 - __exit__ - DEBUG - Time: 55.32 seconds 2025-02-14 14:11:41,566 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:11:41,566 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37615.88 MB 2025-02-14 14:11:41,566 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 50133.78 MB 2025-02-14 14:11:41,566 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12517.90 MB 2025-02-14 14:11:41,566 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 75004.64 MB 2025-02-14 14:11:41,566 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54081.36 MB 2025-02-14 14:11:41,566 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20923.29 MB 2025-02-14 14:11:41,566 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 62651.68 MB 2025-02-14 14:11:41,919 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 14:11:41,919 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 14:11:41,919 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.35 seconds 2025-02-14 14:11:41,919 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:11:41,920 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 50133.78 MB 2025-02-14 14:11:41,920 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34166.40 MB 2025-02-14 14:11:41,920 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -15967.38 MB 2025-02-14 14:11:41,920 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54081.36 MB 2025-02-14 14:11:41,920 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 91498.74 MB 2025-02-14 14:11:41,920 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 37417.39 MB 2025-02-14 14:11:41,920 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 86547.29 MB 2025-02-14 14:11:43,944 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 14:11:43,944 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 14:11:43,944 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.02 seconds 2025-02-14 14:11:43,944 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:11:43,944 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34166.40 MB 2025-02-14 14:11:43,944 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34697.24 MB 2025-02-14 14:11:43,944 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 14:11:43,944 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 91498.74 MB 2025-02-14 14:11:43,944 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36719.03 MB 2025-02-14 14:11:43,944 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -54779.71 MB 2025-02-14 14:11:43,944 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38677.61 MB 2025-02-14 14:11:43,958 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 14:11:43,958 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 14:11:43,958 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 14:11:43,958 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:11:43,958 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34697.24 MB 2025-02-14 14:11:43,958 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36586.78 MB 2025-02-14 14:11:43,958 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 14:11:43,958 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36719.03 MB 2025-02-14 14:11:43,958 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40022.05 MB 2025-02-14 14:11:43,958 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 14:11:43,958 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38004.20 MB 2025-02-14 14:11:44,172 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 14:11:44,172 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 14:11:44,172 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 14:11:44,172 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:11:44,172 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36586.78 MB 2025-02-14 14:11:44,172 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38828.63 MB 2025-02-14 14:11:44,173 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 14:11:44,173 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40022.05 MB 2025-02-14 14:11:44,173 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46628.08 MB 2025-02-14 14:11:44,173 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 14:11:44,173 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44372.91 MB 2025-02-14 14:11:44,173 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 14:11:44,173 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 14:11:44,173 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 14:11:44,173 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:11:44,173 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34697.24 MB 2025-02-14 14:11:44,173 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38828.63 MB 2025-02-14 14:11:44,173 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 14:11:44,173 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36719.03 MB 2025-02-14 14:11:44,173 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46628.08 MB 2025-02-14 14:11:44,173 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9909.04 MB 2025-02-14 14:11:44,173 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44372.91 MB 2025-02-14 14:11:44,350 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 14:11:44,350 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 14:11:44,350 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 14:11:44,350 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:11:44,350 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40362.17 MB 2025-02-14 14:11:44,350 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41129.18 MB 2025-02-14 14:11:44,350 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 14:11:44,350 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46628.08 MB 2025-02-14 14:11:44,350 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47039.12 MB 2025-02-14 14:11:44,350 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 411.04 MB 2025-02-14 14:11:44,350 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41836.96 MB 2025-02-14 14:11:44,369 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 14:11:44,369 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 14:11:44,369 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:11:44,369 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:11:44,370 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41542.06 MB 2025-02-14 14:11:44,370 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41771.36 MB 2025-02-14 14:11:44,370 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.29 MB 2025-02-14 14:11:44,370 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47039.12 MB 2025-02-14 14:11:44,370 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47039.12 MB 2025-02-14 14:11:44,370 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:11:44,370 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41988.62 MB 2025-02-14 14:11:44,371 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 14:11:44,371 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 14:11:44,371 - resource_logging.py:150 - __exit__ - DEBUG - Time: 58.14 seconds 2025-02-14 14:11:44,371 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:11:44,371 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25292.29 MB 2025-02-14 14:11:44,371 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41972.43 MB 2025-02-14 14:11:44,371 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 16680.14 MB 2025-02-14 14:11:44,371 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62679.68 MB 2025-02-14 14:11:44,371 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47039.12 MB 2025-02-14 14:11:44,371 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15640.56 MB 2025-02-14 14:11:44,371 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41988.62 MB 2025-02-14 14:11:44,647 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 14:11:44,647 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 14:11:44,647 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 14:11:44,647 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:11:44,647 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41972.43 MB 2025-02-14 14:11:44,647 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30296.68 MB 2025-02-14 14:11:44,647 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -11675.75 MB 2025-02-14 14:11:44,647 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47039.12 MB 2025-02-14 14:11:44,647 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47039.12 MB 2025-02-14 14:11:44,647 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:11:44,647 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44484.10 MB 2025-02-14 14:11:44,665 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 14:11:44,666 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 14:11:44,671 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 14:11:44,671 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 14:11:44,671 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:11:44,671 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:11:44,671 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30296.68 MB 2025-02-14 14:11:44,671 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38735.37 MB 2025-02-14 14:11:44,671 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8438.69 MB 2025-02-14 14:11:44,671 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47039.12 MB 2025-02-14 14:11:44,671 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51235.52 MB 2025-02-14 14:11:44,672 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4196.40 MB 2025-02-14 14:11:44,672 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38735.37 MB 2025-02-14 14:11:44,840 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 14:11:44,842 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:11:44,842 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 14:11:44,843 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:11:44,843 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 14:11:44,848 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 14:11:44,849 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:11:44,849 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 14:11:44,849 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 14:12:22,126 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:12:22,126 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 14:12:22,130 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 14:12:22,134 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:12:22,134 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 734, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 14:12:22,135 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:12:22,135 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 734, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 14:12:33,532 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 14:12:33,533 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 14:12:33,533 - resource_logging.py:150 - __exit__ - DEBUG - Time: 11.39 seconds 2025-02-14 14:12:33,533 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:12:33,533 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18083.34 MB 2025-02-14 14:12:33,533 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20681.71 MB 2025-02-14 14:12:33,533 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2598.37 MB 2025-02-14 14:12:33,533 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59624.13 MB 2025-02-14 14:12:33,533 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25117.59 MB 2025-02-14 14:12:33,533 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34506.54 MB 2025-02-14 14:12:33,533 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29593.95 MB 2025-02-14 14:12:33,593 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 14:12:33,593 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 14:12:33,593 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 14:12:33,593 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:12:33,593 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20681.71 MB 2025-02-14 14:12:33,593 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19594.74 MB 2025-02-14 14:12:33,593 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1086.97 MB 2025-02-14 14:12:33,593 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25117.59 MB 2025-02-14 14:12:33,593 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32168.21 MB 2025-02-14 14:12:33,593 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7050.63 MB 2025-02-14 14:12:33,593 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29543.56 MB 2025-02-14 14:12:35,512 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 14:12:35,512 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 14:12:35,513 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 14:12:35,513 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:12:35,513 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19594.74 MB 2025-02-14 14:12:35,513 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20125.58 MB 2025-02-14 14:12:35,513 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 14:12:35,513 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32168.21 MB 2025-02-14 14:12:35,513 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24643.63 MB 2025-02-14 14:12:35,513 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7524.58 MB 2025-02-14 14:12:35,513 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24104.91 MB 2025-02-14 14:12:35,527 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 14:12:35,527 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 14:12:35,527 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 14:12:35,527 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:12:35,527 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20125.58 MB 2025-02-14 14:12:35,527 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22015.11 MB 2025-02-14 14:12:35,527 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 14:12:35,527 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24643.63 MB 2025-02-14 14:12:35,527 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26531.07 MB 2025-02-14 14:12:35,527 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 14:12:35,527 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23432.54 MB 2025-02-14 14:12:35,740 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 14:12:35,740 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 14:12:35,740 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 14:12:35,740 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:12:35,741 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22015.11 MB 2025-02-14 14:12:35,741 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24256.97 MB 2025-02-14 14:12:35,741 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 14:12:35,741 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26531.07 MB 2025-02-14 14:12:35,741 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32193.38 MB 2025-02-14 14:12:35,741 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 14:12:35,741 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29801.25 MB 2025-02-14 14:12:35,741 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 14:12:35,741 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 14:12:35,741 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 14:12:35,741 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:12:35,741 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20125.58 MB 2025-02-14 14:12:35,741 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24256.97 MB 2025-02-14 14:12:35,741 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 14:12:35,741 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24643.63 MB 2025-02-14 14:12:35,741 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32193.38 MB 2025-02-14 14:12:35,741 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-14 14:12:35,741 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29801.25 MB 2025-02-14 14:12:35,908 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 14:12:35,908 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 14:12:35,908 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 14:12:35,908 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:12:35,908 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25790.51 MB 2025-02-14 14:12:35,908 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26557.51 MB 2025-02-14 14:12:35,908 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 14:12:35,908 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32193.38 MB 2025-02-14 14:12:35,908 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32608.62 MB 2025-02-14 14:12:35,908 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 14:12:35,908 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27265.30 MB 2025-02-14 14:12:35,927 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 14:12:35,927 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 14:12:35,927 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:12:35,927 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:12:35,927 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26970.40 MB 2025-02-14 14:12:35,927 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27200.41 MB 2025-02-14 14:12:35,927 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 230.01 MB 2025-02-14 14:12:35,927 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32608.62 MB 2025-02-14 14:12:35,927 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32608.62 MB 2025-02-14 14:12:35,927 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:12:35,927 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27405.44 MB 2025-02-14 14:12:35,928 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 14:12:35,928 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 14:12:35,928 - resource_logging.py:150 - __exit__ - DEBUG - Time: 13.79 seconds 2025-02-14 14:12:35,928 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:12:35,928 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15526.02 MB 2025-02-14 14:12:35,928 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27401.48 MB 2025-02-14 14:12:35,928 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11875.46 MB 2025-02-14 14:12:35,928 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59624.13 MB 2025-02-14 14:12:35,928 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32608.62 MB 2025-02-14 14:12:35,928 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27015.51 MB 2025-02-14 14:12:35,928 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27405.44 MB 2025-02-14 14:12:36,196 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 14:12:36,197 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 14:12:36,197 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 14:12:36,197 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:12:36,197 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27401.48 MB 2025-02-14 14:12:36,197 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20530.41 MB 2025-02-14 14:12:36,197 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6871.07 MB 2025-02-14 14:12:36,197 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32608.62 MB 2025-02-14 14:12:36,197 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32608.62 MB 2025-02-14 14:12:36,197 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:12:36,197 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29913.15 MB 2025-02-14 14:12:36,214 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 14:12:36,215 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 14:12:36,221 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 14:12:36,221 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 14:12:36,221 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:12:36,221 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:12:36,221 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20530.41 MB 2025-02-14 14:12:36,221 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28969.43 MB 2025-02-14 14:12:36,221 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 14:12:36,221 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32608.62 MB 2025-02-14 14:12:36,221 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40999.32 MB 2025-02-14 14:12:36,221 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 14:12:36,221 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28969.43 MB 2025-02-14 14:12:36,374 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 14:12:36,375 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:12:36,375 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 14:12:36,376 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:12:36,376 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 14:12:36,381 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 14:12:36,382 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:12:36,382 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 14:12:36,382 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 14:13:04,100 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:13:04,100 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 14:13:04,108 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 14:13:04,115 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:13:04,115 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 808, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 14:13:04,117 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:13:04,117 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 808, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 14:13:16,753 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 14:13:16,753 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 14:13:16,753 - resource_logging.py:150 - __exit__ - DEBUG - Time: 12.63 seconds 2025-02-14 14:13:16,753 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:13:16,753 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18598.98 MB 2025-02-14 14:13:16,753 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21459.49 MB 2025-02-14 14:13:16,753 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2860.52 MB 2025-02-14 14:13:16,753 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53584.33 MB 2025-02-14 14:13:16,753 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26799.51 MB 2025-02-14 14:13:16,753 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26784.83 MB 2025-02-14 14:13:16,753 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30336.08 MB 2025-02-14 14:13:16,806 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 14:13:16,806 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 14:13:16,806 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-14 14:13:16,806 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:13:16,806 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21459.49 MB 2025-02-14 14:13:16,806 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19978.39 MB 2025-02-14 14:13:16,806 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1481.10 MB 2025-02-14 14:13:16,806 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26799.51 MB 2025-02-14 14:13:16,806 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35301.36 MB 2025-02-14 14:13:16,806 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8501.85 MB 2025-02-14 14:13:16,806 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31121.93 MB 2025-02-14 14:13:18,732 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 14:13:18,732 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 14:13:18,732 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 14:13:18,732 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:13:18,732 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19978.39 MB 2025-02-14 14:13:18,732 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20509.23 MB 2025-02-14 14:13:18,732 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 14:13:18,732 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35301.36 MB 2025-02-14 14:13:18,732 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25354.57 MB 2025-02-14 14:13:18,732 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9946.79 MB 2025-02-14 14:13:18,733 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24488.57 MB 2025-02-14 14:13:18,746 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 14:13:18,746 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 14:13:18,746 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 14:13:18,746 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:13:18,746 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20509.23 MB 2025-02-14 14:13:18,746 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22398.77 MB 2025-02-14 14:13:18,746 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 14:13:18,746 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25354.57 MB 2025-02-14 14:13:18,746 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26298.29 MB 2025-02-14 14:13:18,746 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 14:13:18,746 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23816.20 MB 2025-02-14 14:13:18,959 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 14:13:18,959 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 14:13:18,959 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 14:13:18,959 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:13:18,959 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22398.77 MB 2025-02-14 14:13:18,959 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24640.62 MB 2025-02-14 14:13:18,959 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 14:13:18,959 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26298.29 MB 2025-02-14 14:13:18,959 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32432.46 MB 2025-02-14 14:13:18,959 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 14:13:18,959 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30184.90 MB 2025-02-14 14:13:18,960 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 14:13:18,960 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 14:13:18,960 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 14:13:18,960 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:13:18,960 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20509.23 MB 2025-02-14 14:13:18,960 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24640.62 MB 2025-02-14 14:13:18,960 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 14:13:18,960 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25354.57 MB 2025-02-14 14:13:18,960 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32432.46 MB 2025-02-14 14:13:18,960 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7077.89 MB 2025-02-14 14:13:18,960 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30184.90 MB 2025-02-14 14:13:19,129 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 14:13:19,129 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 14:13:19,129 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 14:13:19,129 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:13:19,129 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26174.16 MB 2025-02-14 14:13:19,130 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26941.17 MB 2025-02-14 14:13:19,130 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 14:13:19,130 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32432.46 MB 2025-02-14 14:13:19,130 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32843.50 MB 2025-02-14 14:13:19,130 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 411.04 MB 2025-02-14 14:13:19,130 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27648.95 MB 2025-02-14 14:13:19,149 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 14:13:19,149 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 14:13:19,149 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:13:19,149 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:13:19,149 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27354.06 MB 2025-02-14 14:13:19,149 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27583.14 MB 2025-02-14 14:13:19,149 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.08 MB 2025-02-14 14:13:19,149 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32843.50 MB 2025-02-14 14:13:19,149 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32843.50 MB 2025-02-14 14:13:19,149 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:13:19,149 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27792.00 MB 2025-02-14 14:13:19,150 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 14:13:19,150 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 14:13:19,150 - resource_logging.py:150 - __exit__ - DEBUG - Time: 15.03 seconds 2025-02-14 14:13:19,150 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:13:19,150 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15783.84 MB 2025-02-14 14:13:19,150 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27783.74 MB 2025-02-14 14:13:19,150 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11999.90 MB 2025-02-14 14:13:19,150 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53584.33 MB 2025-02-14 14:13:19,150 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32843.50 MB 2025-02-14 14:13:19,150 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20740.83 MB 2025-02-14 14:13:19,150 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27792.00 MB 2025-02-14 14:13:19,419 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 14:13:19,419 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 14:13:19,419 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 14:13:19,419 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:13:19,419 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27783.74 MB 2025-02-14 14:13:19,419 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20780.99 MB 2025-02-14 14:13:19,419 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7002.75 MB 2025-02-14 14:13:19,419 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32843.50 MB 2025-02-14 14:13:19,419 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32843.50 MB 2025-02-14 14:13:19,419 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:13:19,419 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30289.57 MB 2025-02-14 14:13:19,437 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8143, cut from 8145 2025-02-14 14:13:19,438 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 14:13:19,444 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 14:13:19,444 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 14:13:19,444 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:13:19,444 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:13:19,444 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20780.99 MB 2025-02-14 14:13:19,444 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29200.07 MB 2025-02-14 14:13:19,444 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8419.08 MB 2025-02-14 14:13:19,444 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32843.50 MB 2025-02-14 14:13:19,444 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43308.29 MB 2025-02-14 14:13:19,444 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10464.79 MB 2025-02-14 14:13:19,444 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29200.07 MB 2025-02-14 14:13:19,607 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7935] 2025-02-14 14:13:19,608 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:13:19,608 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 14:13:19,609 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:13:19,609 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 14:13:19,614 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 14:13:19,615 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:13:19,615 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 14:13:19,615 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 14:14:53,945 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:14:53,945 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 14:14:53,950 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 14:14:53,954 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:14:53,954 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 519, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 14:14:53,955 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:14:53,955 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 519, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 14:15:01,944 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 14:15:01,944 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 14:15:01,944 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.98 seconds 2025-02-14 14:15:01,944 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:15:01,944 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16585.18 MB 2025-02-14 14:15:01,944 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18422.29 MB 2025-02-14 14:15:01,944 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1837.11 MB 2025-02-14 14:15:01,944 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51680.12 MB 2025-02-14 14:15:01,944 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22525.51 MB 2025-02-14 14:15:01,944 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29154.61 MB 2025-02-14 14:15:01,944 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27416.31 MB 2025-02-14 14:15:01,979 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 14:15:01,979 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 14:15:01,979 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 14:15:01,979 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:15:01,979 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18422.29 MB 2025-02-14 14:15:01,979 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18475.97 MB 2025-02-14 14:15:01,979 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 53.68 MB 2025-02-14 14:15:01,979 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22525.51 MB 2025-02-14 14:15:01,979 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28695.33 MB 2025-02-14 14:15:01,979 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6169.82 MB 2025-02-14 14:15:01,979 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26009.16 MB 2025-02-14 14:15:03,906 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 14:15:03,907 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 14:15:03,907 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 14:15:03,907 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:15:03,907 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18475.97 MB 2025-02-14 14:15:03,907 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19006.81 MB 2025-02-14 14:15:03,907 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 14:15:03,907 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28695.33 MB 2025-02-14 14:15:03,907 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22001.22 MB 2025-02-14 14:15:03,907 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6694.11 MB 2025-02-14 14:15:03,907 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22987.18 MB 2025-02-14 14:15:03,921 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 14:15:03,921 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 14:15:03,921 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 14:15:03,921 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:15:03,921 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19006.81 MB 2025-02-14 14:15:03,921 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20896.35 MB 2025-02-14 14:15:03,921 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 14:15:03,921 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22001.22 MB 2025-02-14 14:15:03,921 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24832.38 MB 2025-02-14 14:15:03,921 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-14 14:15:03,921 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22313.77 MB 2025-02-14 14:15:04,134 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 14:15:04,134 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 14:15:04,134 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 14:15:04,134 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:15:04,134 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20896.35 MB 2025-02-14 14:15:04,134 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23138.20 MB 2025-02-14 14:15:04,134 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 14:15:04,134 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24832.38 MB 2025-02-14 14:15:04,134 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31438.41 MB 2025-02-14 14:15:04,134 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 14:15:04,134 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28682.48 MB 2025-02-14 14:15:04,135 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 14:15:04,135 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 14:15:04,135 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 14:15:04,135 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:15:04,135 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19006.81 MB 2025-02-14 14:15:04,135 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23138.20 MB 2025-02-14 14:15:04,135 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 14:15:04,135 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22001.22 MB 2025-02-14 14:15:04,135 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31438.41 MB 2025-02-14 14:15:04,135 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9437.18 MB 2025-02-14 14:15:04,135 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28682.48 MB 2025-02-14 14:15:04,298 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 14:15:04,298 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 14:15:04,298 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 14:15:04,298 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:15:04,298 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24671.74 MB 2025-02-14 14:15:04,298 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25438.75 MB 2025-02-14 14:15:04,298 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 14:15:04,298 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31438.41 MB 2025-02-14 14:15:04,298 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31849.45 MB 2025-02-14 14:15:04,298 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 411.04 MB 2025-02-14 14:15:04,298 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26146.53 MB 2025-02-14 14:15:04,317 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 14:15:04,317 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 14:15:04,317 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:15:04,317 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:15:04,317 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25851.63 MB 2025-02-14 14:15:04,317 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26081.69 MB 2025-02-14 14:15:04,317 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 230.05 MB 2025-02-14 14:15:04,317 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31849.45 MB 2025-02-14 14:15:04,317 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31849.45 MB 2025-02-14 14:15:04,317 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:15:04,317 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26241.45 MB 2025-02-14 14:15:04,318 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 14:15:04,318 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 14:15:04,318 - resource_logging.py:150 - __exit__ - DEBUG - Time: 10.36 seconds 2025-02-14 14:15:04,318 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:15:04,318 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14776.94 MB 2025-02-14 14:15:04,318 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26282.76 MB 2025-02-14 14:15:04,318 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11505.82 MB 2025-02-14 14:15:04,318 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51680.12 MB 2025-02-14 14:15:04,318 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31849.45 MB 2025-02-14 14:15:04,318 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19830.67 MB 2025-02-14 14:15:04,318 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26282.76 MB 2025-02-14 14:15:04,585 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 14:15:04,585 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 14:15:04,585 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 14:15:04,585 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:15:04,585 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26282.76 MB 2025-02-14 14:15:04,585 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19781.33 MB 2025-02-14 14:15:04,585 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6501.43 MB 2025-02-14 14:15:04,585 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31849.45 MB 2025-02-14 14:15:04,585 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31849.45 MB 2025-02-14 14:15:04,585 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:15:04,585 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28794.43 MB 2025-02-14 14:15:04,603 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 14:15:04,603 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 14:15:04,609 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 14:15:04,609 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 14:15:04,609 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:15:04,609 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:15:04,609 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19781.33 MB 2025-02-14 14:15:04,609 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28220.36 MB 2025-02-14 14:15:04,609 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 14:15:04,609 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31849.45 MB 2025-02-14 14:15:04,609 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42339.40 MB 2025-02-14 14:15:04,609 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 14:15:04,609 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28220.36 MB 2025-02-14 14:15:04,761 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 14:15:04,762 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:15:04,762 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 14:15:04,763 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:15:04,763 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 14:15:04,768 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 14:15:04,769 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:15:04,769 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 14:15:04,769 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 14:16:08,156 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:16:08,156 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 14:16:08,161 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 14:16:08,165 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:16:08,165 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2214, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 14:16:08,166 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:16:08,166 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2214, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 14:16:42,356 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 14:16:42,356 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 14:16:42,356 - resource_logging.py:150 - __exit__ - DEBUG - Time: 34.18 seconds 2025-02-14 14:16:42,356 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:16:42,356 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28396.21 MB 2025-02-14 14:16:42,356 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36231.44 MB 2025-02-14 14:16:42,356 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7835.22 MB 2025-02-14 14:16:42,356 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54924.41 MB 2025-02-14 14:16:42,356 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41527.80 MB 2025-02-14 14:16:42,356 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13396.61 MB 2025-02-14 14:16:42,356 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45116.15 MB 2025-02-14 14:16:42,586 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 14:16:42,586 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 14:16:42,586 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 14:16:42,586 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:16:42,586 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36231.44 MB 2025-02-14 14:16:42,586 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27287.75 MB 2025-02-14 14:16:42,586 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8943.69 MB 2025-02-14 14:16:42,586 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41527.80 MB 2025-02-14 14:16:42,586 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 70212.65 MB 2025-02-14 14:16:42,586 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 28684.85 MB 2025-02-14 14:16:42,586 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 59323.25 MB 2025-02-14 14:16:44,604 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 14:16:44,604 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 14:16:44,604 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.02 seconds 2025-02-14 14:16:44,604 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:16:44,604 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27287.75 MB 2025-02-14 14:16:44,604 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27818.59 MB 2025-02-14 14:16:44,604 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 14:16:44,604 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 70212.65 MB 2025-02-14 14:16:44,604 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30912.02 MB 2025-02-14 14:16:44,604 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -39300.63 MB 2025-02-14 14:16:44,604 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31798.96 MB 2025-02-14 14:16:44,619 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 14:16:44,619 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 14:16:44,619 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 14:16:44,619 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:16:44,619 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27818.59 MB 2025-02-14 14:16:44,619 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29708.12 MB 2025-02-14 14:16:44,619 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 14:16:44,619 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30912.02 MB 2025-02-14 14:16:44,619 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34215.03 MB 2025-02-14 14:16:44,619 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 14:16:44,619 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31125.55 MB 2025-02-14 14:16:44,851 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 14:16:44,851 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 14:16:44,851 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 14:16:44,851 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:16:44,851 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29708.12 MB 2025-02-14 14:16:44,851 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31949.98 MB 2025-02-14 14:16:44,851 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 14:16:44,851 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34215.03 MB 2025-02-14 14:16:44,851 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40349.20 MB 2025-02-14 14:16:44,851 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 14:16:44,851 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37494.26 MB 2025-02-14 14:16:44,853 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 14:16:44,853 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 14:16:44,853 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.25 seconds 2025-02-14 14:16:44,853 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:16:44,853 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27818.59 MB 2025-02-14 14:16:44,853 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31949.98 MB 2025-02-14 14:16:44,853 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 14:16:44,853 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30912.02 MB 2025-02-14 14:16:44,853 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40349.20 MB 2025-02-14 14:16:44,853 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9437.18 MB 2025-02-14 14:16:44,853 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37494.26 MB 2025-02-14 14:16:45,135 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 14:16:45,135 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 14:16:45,135 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 14:16:45,135 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:16:45,135 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33483.52 MB 2025-02-14 14:16:45,135 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34250.52 MB 2025-02-14 14:16:45,136 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 14:16:45,136 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40349.20 MB 2025-02-14 14:16:45,136 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40762.34 MB 2025-02-14 14:16:45,136 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 14:16:45,136 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34958.31 MB 2025-02-14 14:16:45,168 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 14:16:45,168 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 14:16:45,168 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 14:16:45,168 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:16:45,168 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34663.41 MB 2025-02-14 14:16:45,168 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34891.73 MB 2025-02-14 14:16:45,168 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.32 MB 2025-02-14 14:16:45,168 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40762.34 MB 2025-02-14 14:16:45,168 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40762.34 MB 2025-02-14 14:16:45,168 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:16:45,168 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35107.60 MB 2025-02-14 14:16:45,170 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 14:16:45,170 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 14:16:45,170 - resource_logging.py:150 - __exit__ - DEBUG - Time: 37.00 seconds 2025-02-14 14:16:45,170 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:16:45,171 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20682.46 MB 2025-02-14 14:16:45,171 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35091.97 MB 2025-02-14 14:16:45,171 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14409.51 MB 2025-02-14 14:16:45,171 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54924.41 MB 2025-02-14 14:16:45,171 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40762.34 MB 2025-02-14 14:16:45,171 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14162.07 MB 2025-02-14 14:16:45,171 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35107.60 MB 2025-02-14 14:16:45,462 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 14:16:45,463 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 14:16:45,463 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-14 14:16:45,463 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:16:45,463 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35091.97 MB 2025-02-14 14:16:45,463 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25673.90 MB 2025-02-14 14:16:45,463 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9418.07 MB 2025-02-14 14:16:45,463 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40762.34 MB 2025-02-14 14:16:45,463 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40762.34 MB 2025-02-14 14:16:45,463 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:16:45,463 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37593.19 MB 2025-02-14 14:16:45,482 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8128, cut from 8130 2025-02-14 14:16:45,483 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 14:16:45,490 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 14:16:45,490 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 14:16:45,490 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:16:45,490 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:16:45,490 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25673.90 MB 2025-02-14 14:16:45,490 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34078.98 MB 2025-02-14 14:16:45,490 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8405.08 MB 2025-02-14 14:16:45,490 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40762.34 MB 2025-02-14 14:16:45,490 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49117.40 MB 2025-02-14 14:16:45,490 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8355.05 MB 2025-02-14 14:16:45,491 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34078.98 MB 2025-02-14 14:16:45,736 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7920] 2025-02-14 14:16:45,738 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:16:45,738 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 14:16:45,745 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:16:45,745 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 14:16:45,753 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 14:16:45,755 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:16:45,755 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 14:16:45,755 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 14:16:54,833 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:16:54,834 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 14:16:54,839 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 14:16:54,842 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:16:54,842 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1327, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 14:16:54,843 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:16:54,843 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1327, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 14:17:15,546 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 14:17:15,546 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 14:17:15,546 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.70 seconds 2025-02-14 14:17:15,546 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:17:15,546 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22215.45 MB 2025-02-14 14:17:15,546 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26911.63 MB 2025-02-14 14:17:15,546 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4696.18 MB 2025-02-14 14:17:15,546 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57472.45 MB 2025-02-14 14:17:15,546 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38317.06 MB 2025-02-14 14:17:15,546 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19155.39 MB 2025-02-14 14:17:15,546 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35763.69 MB 2025-02-14 14:17:15,636 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 14:17:15,636 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 14:17:15,637 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 14:17:15,637 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:17:15,637 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26911.63 MB 2025-02-14 14:17:15,637 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22676.51 MB 2025-02-14 14:17:15,637 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4235.12 MB 2025-02-14 14:17:15,637 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38317.06 MB 2025-02-14 14:17:15,637 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47106.23 MB 2025-02-14 14:17:15,637 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8789.16 MB 2025-02-14 14:17:15,637 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40179.18 MB 2025-02-14 14:17:17,601 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 14:17:17,601 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 14:17:17,601 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.96 seconds 2025-02-14 14:17:17,601 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:17:17,601 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22676.51 MB 2025-02-14 14:17:17,601 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23207.35 MB 2025-02-14 14:17:17,601 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 14:17:17,601 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47106.23 MB 2025-02-14 14:17:17,601 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33619.44 MB 2025-02-14 14:17:17,601 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13486.78 MB 2025-02-14 14:17:17,601 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27186.68 MB 2025-02-14 14:17:17,615 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 14:17:17,615 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 14:17:17,615 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 14:17:17,615 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:17:17,615 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23207.35 MB 2025-02-14 14:17:17,615 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25096.89 MB 2025-02-14 14:17:17,615 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 14:17:17,615 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33619.44 MB 2025-02-14 14:17:17,615 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33619.44 MB 2025-02-14 14:17:17,615 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:17:17,615 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26514.31 MB 2025-02-14 14:17:17,832 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 14:17:17,832 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 14:17:17,832 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 14:17:17,832 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:17:17,832 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25096.89 MB 2025-02-14 14:17:17,832 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27338.74 MB 2025-02-14 14:17:17,832 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 14:17:17,832 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33619.44 MB 2025-02-14 14:17:17,832 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35978.74 MB 2025-02-14 14:17:17,832 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2359.30 MB 2025-02-14 14:17:17,832 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32883.02 MB 2025-02-14 14:17:17,832 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 14:17:17,832 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 14:17:17,832 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 14:17:17,833 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:17:17,833 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23207.35 MB 2025-02-14 14:17:17,833 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27338.74 MB 2025-02-14 14:17:17,833 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 14:17:17,833 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33619.44 MB 2025-02-14 14:17:17,833 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35978.74 MB 2025-02-14 14:17:17,833 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2359.30 MB 2025-02-14 14:17:17,833 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32883.02 MB 2025-02-14 14:17:18,007 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 14:17:18,008 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 14:17:18,008 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 14:17:18,008 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:17:18,008 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28872.28 MB 2025-02-14 14:17:18,008 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29639.29 MB 2025-02-14 14:17:18,008 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 14:17:18,008 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35978.74 MB 2025-02-14 14:17:18,008 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36393.98 MB 2025-02-14 14:17:18,008 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 14:17:18,008 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30347.07 MB 2025-02-14 14:17:18,027 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 14:17:18,027 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 14:17:18,027 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:17:18,027 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:17:18,027 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30052.17 MB 2025-02-14 14:17:18,027 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30280.59 MB 2025-02-14 14:17:18,027 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.42 MB 2025-02-14 14:17:18,027 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36393.98 MB 2025-02-14 14:17:18,027 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36393.98 MB 2025-02-14 14:17:18,027 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:17:18,027 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30480.72 MB 2025-02-14 14:17:18,028 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 14:17:18,028 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 14:17:18,028 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.18 seconds 2025-02-14 14:17:18,028 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:17:18,028 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17592.08 MB 2025-02-14 14:17:18,028 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30480.93 MB 2025-02-14 14:17:18,028 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12888.85 MB 2025-02-14 14:17:18,028 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57472.45 MB 2025-02-14 14:17:18,028 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36393.98 MB 2025-02-14 14:17:18,028 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21078.47 MB 2025-02-14 14:17:18,028 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30480.93 MB 2025-02-14 14:17:18,301 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 14:17:18,301 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 14:17:18,301 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 14:17:18,301 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:17:18,301 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30480.93 MB 2025-02-14 14:17:18,301 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22585.04 MB 2025-02-14 14:17:18,301 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7895.89 MB 2025-02-14 14:17:18,301 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36393.98 MB 2025-02-14 14:17:18,301 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36393.98 MB 2025-02-14 14:17:18,301 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:17:18,301 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32983.38 MB 2025-02-14 14:17:18,319 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8132, cut from 8134 2025-02-14 14:17:18,319 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 14:17:18,325 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 14:17:18,325 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 14:17:18,325 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:17:18,325 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:17:18,325 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22585.04 MB 2025-02-14 14:17:18,325 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30994.34 MB 2025-02-14 14:17:18,325 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8409.30 MB 2025-02-14 14:17:18,325 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36393.98 MB 2025-02-14 14:17:18,325 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44753.22 MB 2025-02-14 14:17:18,326 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8359.25 MB 2025-02-14 14:17:18,326 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30994.34 MB 2025-02-14 14:17:18,496 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7924] 2025-02-14 14:17:18,497 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:17:18,497 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 14:17:18,499 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:17:18,499 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 14:17:18,504 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 14:17:18,506 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:17:18,506 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 14:17:18,506 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 14:18:16,636 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:18:16,636 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 14:18:16,641 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 14:18:16,645 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:18:16,645 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 152, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 14:18:16,646 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:18:16,646 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 152, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 14:18:19,021 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 14:18:19,021 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 14:18:19,021 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.37 seconds 2025-02-14 14:18:19,021 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:18:19,021 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14027.87 MB 2025-02-14 14:18:19,021 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14565.79 MB 2025-02-14 14:18:19,021 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 537.92 MB 2025-02-14 14:18:19,021 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53112.47 MB 2025-02-14 14:18:19,021 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 16907.24 MB 2025-02-14 14:18:19,021 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -36205.23 MB 2025-02-14 14:18:19,021 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23500.04 MB 2025-02-14 14:18:19,033 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 14:18:19,033 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 14:18:19,033 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 14:18:19,033 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:18:19,033 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14565.79 MB 2025-02-14 14:18:19,033 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14699.99 MB 2025-02-14 14:18:19,033 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 134.21 MB 2025-02-14 14:18:19,033 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 16907.24 MB 2025-02-14 14:18:19,033 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17383.29 MB 2025-02-14 14:18:19,033 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 476.05 MB 2025-02-14 14:18:19,033 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16476.33 MB 2025-02-14 14:18:19,686 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 14:18:19,686 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 14:18:19,686 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.65 seconds 2025-02-14 14:18:19,686 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:18:19,686 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14699.99 MB 2025-02-14 14:18:19,686 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14877.83 MB 2025-02-14 14:18:19,686 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 177.83 MB 2025-02-14 14:18:19,686 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17383.29 MB 2025-02-14 14:18:19,686 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17383.29 MB 2025-02-14 14:18:19,686 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:18:19,686 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18871.47 MB 2025-02-14 14:18:19,693 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 14:18:19,693 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 14:18:19,693 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 14:18:19,693 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:18:19,693 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14877.76 MB 2025-02-14 14:18:19,693 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15510.60 MB 2025-02-14 14:18:19,693 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 632.84 MB 2025-02-14 14:18:19,693 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17383.29 MB 2025-02-14 14:18:19,693 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17383.29 MB 2025-02-14 14:18:19,693 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:18:19,693 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15985.44 MB 2025-02-14 14:18:19,769 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 14:18:19,769 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 14:18:19,769 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 14:18:19,769 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:18:19,769 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15510.60 MB 2025-02-14 14:18:19,769 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16261.66 MB 2025-02-14 14:18:19,769 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 751.06 MB 2025-02-14 14:18:19,769 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17383.29 MB 2025-02-14 14:18:19,769 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18811.45 MB 2025-02-14 14:18:19,769 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1428.16 MB 2025-02-14 14:18:19,769 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18120.53 MB 2025-02-14 14:18:19,769 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 14:18:19,769 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 14:18:19,769 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 14:18:19,769 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:18:19,769 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14877.76 MB 2025-02-14 14:18:19,769 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16261.66 MB 2025-02-14 14:18:19,769 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1383.90 MB 2025-02-14 14:18:19,769 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17383.29 MB 2025-02-14 14:18:19,769 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18811.45 MB 2025-02-14 14:18:19,770 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1428.16 MB 2025-02-14 14:18:19,770 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18120.53 MB 2025-02-14 14:18:19,828 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 14:18:19,828 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 14:18:19,828 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 14:18:19,828 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:18:19,828 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16775.40 MB 2025-02-14 14:18:19,828 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17032.35 MB 2025-02-14 14:18:19,828 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 256.95 MB 2025-02-14 14:18:19,828 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18811.45 MB 2025-02-14 14:18:19,828 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18949.87 MB 2025-02-14 14:18:19,828 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 138.41 MB 2025-02-14 14:18:19,828 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17282.53 MB 2025-02-14 14:18:19,837 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 14:18:19,837 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 14:18:19,837 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 14:18:19,837 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:18:19,837 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17170.67 MB 2025-02-14 14:18:19,837 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17390.20 MB 2025-02-14 14:18:19,837 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 219.53 MB 2025-02-14 14:18:19,837 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18949.87 MB 2025-02-14 14:18:19,837 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18949.87 MB 2025-02-14 14:18:19,837 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:18:19,837 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17391.01 MB 2025-02-14 14:18:19,839 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 14:18:19,839 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 14:18:19,839 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.19 seconds 2025-02-14 14:18:19,839 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:18:19,839 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13498.29 MB 2025-02-14 14:18:19,839 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14232.89 MB 2025-02-14 14:18:19,839 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 734.60 MB 2025-02-14 14:18:19,839 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53112.47 MB 2025-02-14 14:18:19,839 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18949.87 MB 2025-02-14 14:18:19,839 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34162.61 MB 2025-02-14 14:18:19,839 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17590.96 MB 2025-02-14 14:18:20,105 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 14:18:20,105 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 14:18:20,105 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 14:18:20,106 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:18:20,106 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14232.89 MB 2025-02-14 14:18:20,106 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17242.77 MB 2025-02-14 14:18:20,106 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3009.88 MB 2025-02-14 14:18:20,106 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18949.87 MB 2025-02-14 14:18:20,106 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18949.87 MB 2025-02-14 14:18:20,106 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:18:20,106 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17543.66 MB 2025-02-14 14:18:20,124 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8149, cut from 8151 2025-02-14 14:18:20,124 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 14:18:20,130 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 14:18:20,130 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 14:18:20,130 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:18:20,130 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:18:20,130 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17242.77 MB 2025-02-14 14:18:20,130 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25668.95 MB 2025-02-14 14:18:20,130 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8426.18 MB 2025-02-14 14:18:20,130 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18949.87 MB 2025-02-14 14:18:20,130 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29420.95 MB 2025-02-14 14:18:20,130 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10471.08 MB 2025-02-14 14:18:20,130 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25668.95 MB 2025-02-14 14:18:20,293 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7941] 2025-02-14 14:18:20,294 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:18:20,294 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 14:18:20,295 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:18:20,295 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 14:18:20,300 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 14:18:20,301 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:18:20,301 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 14:18:20,302 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 14:18:58,007 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:18:58,007 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 14:18:58,012 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 14:18:58,016 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:18:58,017 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1288, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 14:18:58,018 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:18:58,018 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1288, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 14:19:17,867 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 14:19:17,867 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 14:19:17,867 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.84 seconds 2025-02-14 14:19:17,867 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:19:17,867 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27343.53 MB 2025-02-14 14:19:17,867 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31902.74 MB 2025-02-14 14:19:17,867 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4559.21 MB 2025-02-14 14:19:17,867 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40007.37 MB 2025-02-14 14:19:17,867 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41542.48 MB 2025-02-14 14:19:17,867 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1535.12 MB 2025-02-14 14:19:17,867 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40891.77 MB 2025-02-14 14:19:17,942 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 14:19:17,942 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 14:19:17,942 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 14:19:17,942 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:19:17,942 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31902.74 MB 2025-02-14 14:19:17,942 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27873.60 MB 2025-02-14 14:19:17,942 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4029.14 MB 2025-02-14 14:19:17,942 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41542.48 MB 2025-02-14 14:19:17,942 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50591.69 MB 2025-02-14 14:19:17,942 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9049.21 MB 2025-02-14 14:19:17,942 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45537.27 MB 2025-02-14 14:19:19,872 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 14:19:19,872 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 14:19:19,872 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 14:19:19,872 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:19:19,872 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27873.60 MB 2025-02-14 14:19:19,872 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28404.44 MB 2025-02-14 14:19:19,872 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 14:19:19,872 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50591.69 MB 2025-02-14 14:19:19,872 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32795.26 MB 2025-02-14 14:19:19,872 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17796.43 MB 2025-02-14 14:19:19,872 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32384.81 MB 2025-02-14 14:19:19,886 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 14:19:19,886 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 14:19:19,886 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 14:19:19,886 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:19:19,886 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28404.44 MB 2025-02-14 14:19:19,886 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30293.86 MB 2025-02-14 14:19:19,886 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.42 MB 2025-02-14 14:19:19,886 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32795.26 MB 2025-02-14 14:19:19,886 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35626.42 MB 2025-02-14 14:19:19,886 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-14 14:19:19,886 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31711.29 MB 2025-02-14 14:19:20,105 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 14:19:20,105 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 14:19:20,105 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 14:19:20,105 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:19:20,105 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30293.86 MB 2025-02-14 14:19:20,105 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27135.88 MB 2025-02-14 14:19:20,105 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3157.98 MB 2025-02-14 14:19:20,105 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35626.42 MB 2025-02-14 14:19:20,105 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37513.85 MB 2025-02-14 14:19:20,105 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 14:19:20,105 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32680.16 MB 2025-02-14 14:19:20,106 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 14:19:20,106 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 14:19:20,106 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 14:19:20,106 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:19:20,106 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28404.44 MB 2025-02-14 14:19:20,106 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27135.88 MB 2025-02-14 14:19:20,106 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1268.56 MB 2025-02-14 14:19:20,106 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32795.26 MB 2025-02-14 14:19:20,106 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37513.85 MB 2025-02-14 14:19:20,106 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 14:19:20,106 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32680.16 MB 2025-02-14 14:19:20,280 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 14:19:20,280 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 14:19:20,280 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 14:19:20,280 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:19:20,280 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28669.42 MB 2025-02-14 14:19:20,280 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29436.42 MB 2025-02-14 14:19:20,280 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 14:19:20,280 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37513.85 MB 2025-02-14 14:19:20,280 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37929.09 MB 2025-02-14 14:19:20,280 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 14:19:20,280 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30144.21 MB 2025-02-14 14:19:20,299 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 14:19:20,299 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 14:19:20,299 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:19:20,299 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:19:20,299 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29849.31 MB 2025-02-14 14:19:20,299 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30078.22 MB 2025-02-14 14:19:20,299 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.91 MB 2025-02-14 14:19:20,299 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37929.09 MB 2025-02-14 14:19:20,299 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37929.09 MB 2025-02-14 14:19:20,299 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:19:20,299 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30317.99 MB 2025-02-14 14:19:20,300 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 14:19:20,301 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 14:19:20,301 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.28 seconds 2025-02-14 14:19:20,301 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:19:20,301 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22856.04 MB 2025-02-14 14:19:20,301 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30278.48 MB 2025-02-14 14:19:20,301 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7422.45 MB 2025-02-14 14:19:20,301 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37796.97 MB 2025-02-14 14:19:20,301 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37929.09 MB 2025-02-14 14:19:20,301 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 132.12 MB 2025-02-14 14:19:20,301 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30317.99 MB 2025-02-14 14:19:20,569 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 14:19:20,570 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 14:19:20,570 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 14:19:20,570 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:19:20,570 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30278.48 MB 2025-02-14 14:19:20,570 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22448.02 MB 2025-02-14 14:19:20,570 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7830.46 MB 2025-02-14 14:19:20,570 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37929.09 MB 2025-02-14 14:19:20,570 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37929.09 MB 2025-02-14 14:19:20,570 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:19:20,570 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32780.01 MB 2025-02-14 14:19:20,587 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8129, cut from 8131 2025-02-14 14:19:20,587 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 14:19:20,593 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 14:19:20,594 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 14:19:20,594 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:19:20,594 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:19:20,594 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22448.02 MB 2025-02-14 14:19:20,594 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30853.13 MB 2025-02-14 14:19:20,594 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8405.11 MB 2025-02-14 14:19:20,594 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37929.09 MB 2025-02-14 14:19:20,594 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42106.62 MB 2025-02-14 14:19:20,594 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4177.53 MB 2025-02-14 14:19:20,594 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30853.13 MB 2025-02-14 14:19:20,756 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7921] 2025-02-14 14:19:20,757 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:19:20,757 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 14:19:20,758 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:19:20,758 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 14:19:20,763 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 14:19:20,764 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:19:20,764 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 14:19:20,764 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 14:20:27,782 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:20:27,782 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 14:20:27,790 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 14:20:27,797 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:20:27,797 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 692, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 14:20:27,799 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:20:27,799 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 692, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 14:20:38,520 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 14:20:38,520 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 14:20:38,520 - resource_logging.py:150 - __exit__ - DEBUG - Time: 10.71 seconds 2025-02-14 14:20:38,520 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:20:38,520 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17790.67 MB 2025-02-14 14:20:38,520 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20240.15 MB 2025-02-14 14:20:38,520 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2449.47 MB 2025-02-14 14:20:38,520 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54641.30 MB 2025-02-14 14:20:38,520 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25755.12 MB 2025-02-14 14:20:38,520 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28886.17 MB 2025-02-14 14:20:38,520 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29074.79 MB 2025-02-14 14:20:38,563 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 14:20:38,563 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 14:20:38,563 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-14 14:20:38,563 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:20:38,563 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20240.15 MB 2025-02-14 14:20:38,563 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19375.34 MB 2025-02-14 14:20:38,563 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -864.80 MB 2025-02-14 14:20:38,563 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25755.12 MB 2025-02-14 14:20:38,563 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31694.26 MB 2025-02-14 14:20:38,563 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5939.13 MB 2025-02-14 14:20:38,563 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28629.70 MB 2025-02-14 14:20:40,484 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 14:20:40,484 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 14:20:40,484 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 14:20:40,484 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:20:40,484 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19375.34 MB 2025-02-14 14:20:40,484 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19906.18 MB 2025-02-14 14:20:40,484 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 14:20:40,484 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31694.26 MB 2025-02-14 14:20:40,484 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22510.83 MB 2025-02-14 14:20:40,484 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9183.43 MB 2025-02-14 14:20:40,484 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23886.56 MB 2025-02-14 14:20:40,498 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 14:20:40,498 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 14:20:40,498 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 14:20:40,498 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:20:40,498 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19906.18 MB 2025-02-14 14:20:40,498 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21795.72 MB 2025-02-14 14:20:40,498 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 14:20:40,498 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22510.83 MB 2025-02-14 14:20:40,498 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25341.98 MB 2025-02-14 14:20:40,498 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-14 14:20:40,498 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23213.15 MB 2025-02-14 14:20:40,712 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 14:20:40,712 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 14:20:40,712 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 14:20:40,712 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:20:40,712 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21795.72 MB 2025-02-14 14:20:40,712 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24038.62 MB 2025-02-14 14:20:40,712 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2242.90 MB 2025-02-14 14:20:40,712 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25341.98 MB 2025-02-14 14:20:40,712 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31713.13 MB 2025-02-14 14:20:40,712 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6371.15 MB 2025-02-14 14:20:40,712 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29582.90 MB 2025-02-14 14:20:40,713 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 14:20:40,713 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 14:20:40,713 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 14:20:40,713 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:20:40,713 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19906.18 MB 2025-02-14 14:20:40,713 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24038.62 MB 2025-02-14 14:20:40,713 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4132.44 MB 2025-02-14 14:20:40,713 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22510.83 MB 2025-02-14 14:20:40,713 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31713.13 MB 2025-02-14 14:20:40,713 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9202.30 MB 2025-02-14 14:20:40,713 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29582.90 MB 2025-02-14 14:20:40,888 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 14:20:40,888 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 14:20:40,888 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 14:20:40,888 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:20:40,888 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25572.17 MB 2025-02-14 14:20:40,888 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26339.17 MB 2025-02-14 14:20:40,888 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 14:20:40,888 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31713.13 MB 2025-02-14 14:20:40,888 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32130.47 MB 2025-02-14 14:20:40,888 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 14:20:40,888 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27046.96 MB 2025-02-14 14:20:40,907 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 14:20:40,907 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 14:20:40,907 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:20:40,907 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:20:40,907 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26752.06 MB 2025-02-14 14:20:40,907 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26980.30 MB 2025-02-14 14:20:40,907 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.25 MB 2025-02-14 14:20:40,907 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32130.47 MB 2025-02-14 14:20:40,907 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32130.47 MB 2025-02-14 14:20:40,907 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:20:40,907 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27170.36 MB 2025-02-14 14:20:40,908 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 14:20:40,908 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 14:20:40,908 - resource_logging.py:150 - __exit__ - DEBUG - Time: 13.11 seconds 2025-02-14 14:20:40,908 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:20:40,908 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15379.69 MB 2025-02-14 14:20:40,908 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27180.47 MB 2025-02-14 14:20:40,908 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11800.78 MB 2025-02-14 14:20:40,908 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54641.30 MB 2025-02-14 14:20:40,908 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32130.47 MB 2025-02-14 14:20:40,909 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22510.83 MB 2025-02-14 14:20:40,909 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27180.47 MB 2025-02-14 14:20:41,176 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 14:20:41,176 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 14:20:41,176 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 14:20:41,176 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:20:41,176 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27180.47 MB 2025-02-14 14:20:41,176 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20369.98 MB 2025-02-14 14:20:41,176 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6810.48 MB 2025-02-14 14:20:41,176 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32130.47 MB 2025-02-14 14:20:41,176 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32130.47 MB 2025-02-14 14:20:41,176 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:20:41,176 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29680.77 MB 2025-02-14 14:20:41,194 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8125, cut from 8127 2025-02-14 14:20:41,194 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 14:20:41,200 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 14:20:41,200 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 14:20:41,200 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:20:41,200 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:20:41,200 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20369.98 MB 2025-02-14 14:20:41,200 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28770.92 MB 2025-02-14 14:20:41,200 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8400.94 MB 2025-02-14 14:20:41,200 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32130.47 MB 2025-02-14 14:20:41,200 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40483.42 MB 2025-02-14 14:20:41,200 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8352.96 MB 2025-02-14 14:20:41,201 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28770.92 MB 2025-02-14 14:20:41,365 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7917] 2025-02-14 14:20:41,366 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:20:41,366 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 14:20:41,367 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:20:41,367 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 14:20:41,372 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 14:20:41,373 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:20:41,373 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 14:20:41,373 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 14:20:50,214 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:20:50,214 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 14:20:50,219 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 14:20:50,222 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:20:50,222 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1374, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 14:20:50,223 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:20:50,223 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1374, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 14:21:11,544 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 14:21:11,544 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 14:21:11,544 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.31 seconds 2025-02-14 14:21:11,544 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:21:11,544 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22542.96 MB 2025-02-14 14:21:11,544 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27406.25 MB 2025-02-14 14:21:11,544 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4863.30 MB 2025-02-14 14:21:11,544 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53011.81 MB 2025-02-14 14:21:11,544 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38476.45 MB 2025-02-14 14:21:11,544 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14535.36 MB 2025-02-14 14:21:11,544 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36317.69 MB 2025-02-14 14:21:11,620 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 14:21:11,620 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 14:21:11,620 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 14:21:11,620 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:21:11,620 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27406.25 MB 2025-02-14 14:21:11,620 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22920.85 MB 2025-02-14 14:21:11,620 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4485.41 MB 2025-02-14 14:21:11,620 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38476.45 MB 2025-02-14 14:21:11,620 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46823.11 MB 2025-02-14 14:21:11,620 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8346.66 MB 2025-02-14 14:21:11,620 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40024.46 MB 2025-02-14 14:21:13,544 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 14:21:13,544 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 14:21:13,544 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 14:21:13,544 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:21:13,544 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22920.85 MB 2025-02-14 14:21:13,544 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23451.69 MB 2025-02-14 14:21:13,544 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 14:21:13,544 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46823.11 MB 2025-02-14 14:21:13,544 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33613.15 MB 2025-02-14 14:21:13,544 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13209.96 MB 2025-02-14 14:21:13,544 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27431.02 MB 2025-02-14 14:21:13,557 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 14:21:13,557 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 14:21:13,557 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 14:21:13,557 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:21:13,557 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23451.69 MB 2025-02-14 14:21:13,557 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25341.22 MB 2025-02-14 14:21:13,557 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 14:21:13,557 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33613.15 MB 2025-02-14 14:21:13,557 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33613.15 MB 2025-02-14 14:21:13,557 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:21:13,557 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26758.65 MB 2025-02-14 14:21:13,769 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 14:21:13,769 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 14:21:13,769 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 14:21:13,769 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:21:13,769 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25341.22 MB 2025-02-14 14:21:13,769 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27583.08 MB 2025-02-14 14:21:13,769 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 14:21:13,769 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33613.15 MB 2025-02-14 14:21:13,769 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37388.03 MB 2025-02-14 14:21:13,769 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 14:21:13,769 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33127.36 MB 2025-02-14 14:21:13,770 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 14:21:13,770 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 14:21:13,770 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 14:21:13,770 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:21:13,770 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23451.69 MB 2025-02-14 14:21:13,770 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27583.08 MB 2025-02-14 14:21:13,770 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 14:21:13,770 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33613.15 MB 2025-02-14 14:21:13,770 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37388.03 MB 2025-02-14 14:21:13,770 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 14:21:13,770 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33127.36 MB 2025-02-14 14:21:13,938 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 14:21:13,938 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 14:21:13,938 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 14:21:13,938 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:21:13,938 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29116.62 MB 2025-02-14 14:21:13,938 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29883.62 MB 2025-02-14 14:21:13,938 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 14:21:13,938 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37388.03 MB 2025-02-14 14:21:13,938 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37803.26 MB 2025-02-14 14:21:13,938 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 14:21:13,938 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30591.41 MB 2025-02-14 14:21:13,957 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 14:21:13,957 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 14:21:13,957 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:21:13,957 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:21:13,957 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30296.51 MB 2025-02-14 14:21:13,957 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30525.08 MB 2025-02-14 14:21:13,957 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.57 MB 2025-02-14 14:21:13,957 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37803.26 MB 2025-02-14 14:21:13,957 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37803.26 MB 2025-02-14 14:21:13,957 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:21:13,957 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30735.64 MB 2025-02-14 14:21:13,958 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 14:21:13,958 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 14:21:13,958 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.73 seconds 2025-02-14 14:21:13,958 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:21:13,958 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17755.83 MB 2025-02-14 14:21:13,958 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30725.56 MB 2025-02-14 14:21:13,958 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12969.73 MB 2025-02-14 14:21:13,958 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53011.81 MB 2025-02-14 14:21:13,958 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37803.26 MB 2025-02-14 14:21:13,958 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15208.55 MB 2025-02-14 14:21:13,958 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30735.64 MB 2025-02-14 14:21:14,226 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 14:21:14,226 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 14:21:14,226 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 14:21:14,226 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:21:14,226 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30725.56 MB 2025-02-14 14:21:14,226 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22751.08 MB 2025-02-14 14:21:14,226 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7974.48 MB 2025-02-14 14:21:14,226 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37803.26 MB 2025-02-14 14:21:14,226 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37803.26 MB 2025-02-14 14:21:14,226 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:21:14,226 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33229.86 MB 2025-02-14 14:21:14,244 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8138, cut from 8140 2025-02-14 14:21:14,244 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 14:21:14,250 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 14:21:14,250 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 14:21:14,250 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:21:14,251 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:21:14,251 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22751.08 MB 2025-02-14 14:21:14,251 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31165.06 MB 2025-02-14 14:21:14,251 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8413.98 MB 2025-02-14 14:21:14,251 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37803.26 MB 2025-02-14 14:21:14,251 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46168.80 MB 2025-02-14 14:21:14,251 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8365.54 MB 2025-02-14 14:21:14,251 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31165.06 MB 2025-02-14 14:21:14,413 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7930] 2025-02-14 14:21:14,414 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:21:14,414 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 14:21:14,415 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:21:14,415 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 14:21:14,420 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 14:21:14,421 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:21:14,421 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 14:21:14,421 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 14:22:02,369 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:22:02,369 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 14:22:02,374 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 14:22:02,378 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:22:02,378 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 161, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 14:22:02,379 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:22:02,379 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 161, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 14:22:04,876 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 14:22:04,876 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 14:22:04,876 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.49 seconds 2025-02-14 14:22:04,876 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:22:04,876 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14090.58 MB 2025-02-14 14:22:04,876 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14660.35 MB 2025-02-14 14:22:04,876 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 569.77 MB 2025-02-14 14:22:04,876 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58716.06 MB 2025-02-14 14:22:04,876 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21084.77 MB 2025-02-14 14:22:04,876 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -37631.30 MB 2025-02-14 14:22:04,876 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23561.95 MB 2025-02-14 14:22:04,888 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 14:22:04,888 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 14:22:04,888 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 14:22:04,888 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:22:04,888 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14660.35 MB 2025-02-14 14:22:04,888 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14936.40 MB 2025-02-14 14:22:04,888 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 276.05 MB 2025-02-14 14:22:04,888 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21084.77 MB 2025-02-14 14:22:04,888 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21084.77 MB 2025-02-14 14:22:04,888 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:22:04,888 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16925.37 MB 2025-02-14 14:22:05,658 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 14:22:05,658 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 14:22:05,658 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.77 seconds 2025-02-14 14:22:05,658 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:22:05,658 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14936.40 MB 2025-02-14 14:22:05,658 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15150.07 MB 2025-02-14 14:22:05,658 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 213.66 MB 2025-02-14 14:22:05,658 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21084.77 MB 2025-02-14 14:22:05,658 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21084.77 MB 2025-02-14 14:22:05,658 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:22:05,658 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19106.84 MB 2025-02-14 14:22:05,666 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 14:22:05,666 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 14:22:05,666 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 14:22:05,666 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:22:05,666 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15150.00 MB 2025-02-14 14:22:05,666 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15910.35 MB 2025-02-14 14:22:05,666 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 760.35 MB 2025-02-14 14:22:05,666 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21084.77 MB 2025-02-14 14:22:05,666 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21084.77 MB 2025-02-14 14:22:05,666 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:22:05,666 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16480.87 MB 2025-02-14 14:22:05,755 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 14:22:05,755 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 14:22:05,755 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 14:22:05,755 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:22:05,755 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15910.35 MB 2025-02-14 14:22:05,755 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16812.74 MB 2025-02-14 14:22:05,755 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 902.39 MB 2025-02-14 14:22:05,755 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21084.77 MB 2025-02-14 14:22:05,755 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21084.77 MB 2025-02-14 14:22:05,755 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:22:05,755 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19044.27 MB 2025-02-14 14:22:05,756 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 14:22:05,756 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 14:22:05,756 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 14:22:05,756 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:22:05,756 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15150.00 MB 2025-02-14 14:22:05,756 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16812.74 MB 2025-02-14 14:22:05,756 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1662.74 MB 2025-02-14 14:22:05,756 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21084.77 MB 2025-02-14 14:22:05,756 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21084.77 MB 2025-02-14 14:22:05,756 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:22:05,756 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19044.27 MB 2025-02-14 14:22:05,827 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 14:22:05,828 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 14:22:05,828 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 14:22:05,828 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:22:05,828 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17429.99 MB 2025-02-14 14:22:05,828 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17738.71 MB 2025-02-14 14:22:05,828 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 308.72 MB 2025-02-14 14:22:05,828 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21084.77 MB 2025-02-14 14:22:05,828 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21248.34 MB 2025-02-14 14:22:05,828 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 163.58 MB 2025-02-14 14:22:05,828 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18033.17 MB 2025-02-14 14:22:05,837 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 14:22:05,837 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 14:22:05,837 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 14:22:05,837 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:22:05,837 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17904.90 MB 2025-02-14 14:22:05,837 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18134.55 MB 2025-02-14 14:22:05,837 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.64 MB 2025-02-14 14:22:05,837 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21248.34 MB 2025-02-14 14:22:05,837 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21248.34 MB 2025-02-14 14:22:05,837 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:22:05,837 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18160.32 MB 2025-02-14 14:22:05,838 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 14:22:05,838 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 14:22:05,838 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.46 seconds 2025-02-14 14:22:05,838 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:22:05,838 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13529.64 MB 2025-02-14 14:22:05,838 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18335.62 MB 2025-02-14 14:22:05,838 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4805.98 MB 2025-02-14 14:22:05,838 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58716.06 MB 2025-02-14 14:22:05,838 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21248.34 MB 2025-02-14 14:22:05,838 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -37467.72 MB 2025-02-14 14:22:05,838 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18335.62 MB 2025-02-14 14:22:06,104 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 14:22:06,105 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 14:22:06,105 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 14:22:06,105 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:22:06,105 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18335.62 MB 2025-02-14 14:22:06,105 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17406.12 MB 2025-02-14 14:22:06,105 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -929.50 MB 2025-02-14 14:22:06,105 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21248.34 MB 2025-02-14 14:22:06,105 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21248.34 MB 2025-02-14 14:22:06,105 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:22:06,105 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19139.35 MB 2025-02-14 14:22:06,123 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 14:22:06,123 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2.'] 2025-02-14 14:22:06,129 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 14:22:06,129 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 14:22:06,129 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:22:06,129 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:22:06,129 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17406.12 MB 2025-02-14 14:22:06,129 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25845.14 MB 2025-02-14 14:22:06,129 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 14:22:06,129 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21248.34 MB 2025-02-14 14:22:06,129 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31738.30 MB 2025-02-14 14:22:06,129 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 14:22:06,129 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25845.14 MB 2025-02-14 14:22:06,295 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 14:22:06,297 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:22:06,297 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 14:22:06,298 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:22:06,298 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 14:22:06,302 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 14:22:06,303 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:22:06,303 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 14:22:06,304 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2.'] 2025-02-14 14:23:01,335 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:23:01,335 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 14:23:01,340 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 14:23:01,344 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:23:01,344 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1015, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 14:23:01,345 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:23:01,345 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1015, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 14:23:16,882 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 14:23:16,882 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 14:23:16,882 - resource_logging.py:150 - __exit__ - DEBUG - Time: 15.53 seconds 2025-02-14 14:23:16,882 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:23:16,882 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20041.39 MB 2025-02-14 14:23:16,882 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23633.81 MB 2025-02-14 14:23:16,882 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3592.42 MB 2025-02-14 14:23:16,882 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44323.31 MB 2025-02-14 14:23:16,882 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30968.64 MB 2025-02-14 14:23:16,882 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13354.66 MB 2025-02-14 14:23:16,882 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32457.97 MB 2025-02-14 14:23:16,942 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 14:23:16,942 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 14:23:16,942 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 14:23:16,942 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:23:16,942 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23633.81 MB 2025-02-14 14:23:16,942 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21054.52 MB 2025-02-14 14:23:16,942 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2579.29 MB 2025-02-14 14:23:16,942 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30968.64 MB 2025-02-14 14:23:16,942 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40498.10 MB 2025-02-14 14:23:16,942 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9529.46 MB 2025-02-14 14:23:16,942 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34576.31 MB 2025-02-14 14:23:18,847 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 14:23:18,847 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 14:23:18,847 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.90 seconds 2025-02-14 14:23:18,847 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:23:18,847 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21054.52 MB 2025-02-14 14:23:18,847 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21585.36 MB 2025-02-14 14:23:18,847 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 14:23:18,847 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40498.10 MB 2025-02-14 14:23:18,847 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28791.80 MB 2025-02-14 14:23:18,847 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11706.30 MB 2025-02-14 14:23:18,847 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25564.69 MB 2025-02-14 14:23:18,863 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 14:23:18,863 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 14:23:18,863 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 14:23:18,863 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:23:18,863 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21585.36 MB 2025-02-14 14:23:18,863 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23474.89 MB 2025-02-14 14:23:18,863 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 14:23:18,863 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28791.80 MB 2025-02-14 14:23:18,863 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29735.52 MB 2025-02-14 14:23:18,863 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 14:23:18,863 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24892.32 MB 2025-02-14 14:23:19,069 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 14:23:19,069 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 14:23:19,069 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 14:23:19,069 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:23:19,069 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23474.89 MB 2025-02-14 14:23:19,069 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25716.75 MB 2025-02-14 14:23:19,069 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 14:23:19,069 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29735.52 MB 2025-02-14 14:23:19,069 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34925.97 MB 2025-02-14 14:23:19,069 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 14:23:19,069 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31261.03 MB 2025-02-14 14:23:19,069 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 14:23:19,069 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 14:23:19,070 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 14:23:19,070 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:23:19,070 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21585.36 MB 2025-02-14 14:23:19,070 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25716.75 MB 2025-02-14 14:23:19,070 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 14:23:19,070 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28791.80 MB 2025-02-14 14:23:19,070 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34925.97 MB 2025-02-14 14:23:19,070 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 14:23:19,070 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31261.03 MB 2025-02-14 14:23:19,234 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 14:23:19,234 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 14:23:19,234 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 14:23:19,234 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:23:19,234 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27250.29 MB 2025-02-14 14:23:19,234 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28017.29 MB 2025-02-14 14:23:19,234 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 14:23:19,234 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34925.97 MB 2025-02-14 14:23:19,234 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35343.30 MB 2025-02-14 14:23:19,234 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 14:23:19,234 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28725.08 MB 2025-02-14 14:23:19,254 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 14:23:19,254 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 14:23:19,254 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:23:19,254 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:23:19,254 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28430.18 MB 2025-02-14 14:23:19,254 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28658.65 MB 2025-02-14 14:23:19,254 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.47 MB 2025-02-14 14:23:19,254 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35343.30 MB 2025-02-14 14:23:19,254 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35343.30 MB 2025-02-14 14:23:19,254 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:23:19,254 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28881.07 MB 2025-02-14 14:23:19,255 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 14:23:19,255 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 14:23:19,255 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.91 seconds 2025-02-14 14:23:19,255 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:23:19,255 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16505.05 MB 2025-02-14 14:23:19,255 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28859.04 MB 2025-02-14 14:23:19,255 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12353.99 MB 2025-02-14 14:23:19,255 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44323.31 MB 2025-02-14 14:23:19,255 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35343.30 MB 2025-02-14 14:23:19,255 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8980.00 MB 2025-02-14 14:23:19,255 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28881.07 MB 2025-02-14 14:23:19,524 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 14:23:19,524 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 14:23:19,524 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 14:23:19,524 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:23:19,524 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28859.04 MB 2025-02-14 14:23:19,524 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21498.77 MB 2025-02-14 14:23:19,524 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7360.27 MB 2025-02-14 14:23:19,524 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35343.30 MB 2025-02-14 14:23:19,524 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35343.30 MB 2025-02-14 14:23:19,524 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:23:19,524 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31362.10 MB 2025-02-14 14:23:19,542 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8134, cut from 8136 2025-02-14 14:23:19,542 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 14:23:19,549 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 14:23:19,549 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 14:23:19,549 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:23:19,549 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:23:19,549 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21498.77 MB 2025-02-14 14:23:19,549 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29908.58 MB 2025-02-14 14:23:19,549 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8409.81 MB 2025-02-14 14:23:19,549 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35343.30 MB 2025-02-14 14:23:19,549 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43704.65 MB 2025-02-14 14:23:19,549 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8361.35 MB 2025-02-14 14:23:19,549 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29908.58 MB 2025-02-14 14:23:19,699 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7926] 2025-02-14 14:23:19,700 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:23:19,700 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 14:23:19,701 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:23:19,701 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 14:23:19,706 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 14:23:19,707 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:23:19,707 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 14:23:19,707 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 14:24:14,662 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:24:14,663 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 14:24:14,677 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 14:24:14,683 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:24:14,683 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1449, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 14:24:14,685 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:24:14,685 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1449, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 14:24:37,080 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 14:24:37,080 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 14:24:37,080 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.39 seconds 2025-02-14 14:24:37,080 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:24:37,080 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23065.57 MB 2025-02-14 14:24:37,080 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28193.50 MB 2025-02-14 14:24:37,080 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5127.93 MB 2025-02-14 14:24:37,080 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56245.62 MB 2025-02-14 14:24:37,080 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38751.17 MB 2025-02-14 14:24:37,080 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17494.44 MB 2025-02-14 14:24:37,080 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37066.79 MB 2025-02-14 14:24:37,157 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 14:24:37,157 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 14:24:37,157 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 14:24:37,157 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:24:37,157 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28193.50 MB 2025-02-14 14:24:37,157 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23310.75 MB 2025-02-14 14:24:37,157 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4882.75 MB 2025-02-14 14:24:37,157 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38751.17 MB 2025-02-14 14:24:37,157 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46326.09 MB 2025-02-14 14:24:37,157 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7574.91 MB 2025-02-14 14:24:37,158 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40021.87 MB 2025-02-14 14:24:39,074 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 14:24:39,074 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 14:24:39,074 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 14:24:39,075 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:24:39,075 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23310.75 MB 2025-02-14 14:24:39,075 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23841.59 MB 2025-02-14 14:24:39,075 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 14:24:39,075 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46326.09 MB 2025-02-14 14:24:39,075 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29446.11 MB 2025-02-14 14:24:39,075 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16879.98 MB 2025-02-14 14:24:39,075 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27820.92 MB 2025-02-14 14:24:39,090 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 14:24:39,090 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 14:24:39,090 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 14:24:39,090 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:24:39,090 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23841.59 MB 2025-02-14 14:24:39,090 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25731.13 MB 2025-02-14 14:24:39,090 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 14:24:39,090 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29446.11 MB 2025-02-14 14:24:39,090 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30389.83 MB 2025-02-14 14:24:39,090 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 14:24:39,090 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27148.55 MB 2025-02-14 14:24:39,301 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 14:24:39,301 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 14:24:39,301 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 14:24:39,301 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:24:39,301 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25731.13 MB 2025-02-14 14:24:39,301 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27972.98 MB 2025-02-14 14:24:39,301 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 14:24:39,301 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30389.83 MB 2025-02-14 14:24:39,301 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36052.14 MB 2025-02-14 14:24:39,301 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 14:24:39,301 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33517.26 MB 2025-02-14 14:24:39,302 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 14:24:39,302 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 14:24:39,302 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 14:24:39,302 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:24:39,302 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23841.59 MB 2025-02-14 14:24:39,302 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27972.98 MB 2025-02-14 14:24:39,302 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 14:24:39,302 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29446.11 MB 2025-02-14 14:24:39,302 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36052.14 MB 2025-02-14 14:24:39,302 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 14:24:39,302 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33517.26 MB 2025-02-14 14:24:39,472 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 14:24:39,472 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 14:24:39,472 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 14:24:39,472 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:24:39,472 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29506.52 MB 2025-02-14 14:24:39,472 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30273.53 MB 2025-02-14 14:24:39,472 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 14:24:39,472 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36052.14 MB 2025-02-14 14:24:39,472 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36469.47 MB 2025-02-14 14:24:39,472 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 14:24:39,472 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30981.31 MB 2025-02-14 14:24:39,492 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 14:24:39,492 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 14:24:39,492 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:24:39,492 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:24:39,492 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30686.41 MB 2025-02-14 14:24:39,492 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30914.66 MB 2025-02-14 14:24:39,492 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.25 MB 2025-02-14 14:24:39,492 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36469.47 MB 2025-02-14 14:24:39,492 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36469.47 MB 2025-02-14 14:24:39,492 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:24:39,492 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31135.73 MB 2025-02-14 14:24:39,493 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 14:24:39,493 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 14:24:39,493 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.81 seconds 2025-02-14 14:24:39,493 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:24:39,493 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18017.14 MB 2025-02-14 14:24:39,493 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31114.83 MB 2025-02-14 14:24:39,493 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13097.69 MB 2025-02-14 14:24:39,493 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56245.62 MB 2025-02-14 14:24:39,493 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36469.47 MB 2025-02-14 14:24:39,493 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19776.14 MB 2025-02-14 14:24:39,493 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31135.73 MB 2025-02-14 14:24:39,762 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 14:24:39,762 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 14:24:39,762 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 14:24:39,762 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:24:39,762 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31114.83 MB 2025-02-14 14:24:39,762 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23007.43 MB 2025-02-14 14:24:39,762 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8107.39 MB 2025-02-14 14:24:39,762 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36469.47 MB 2025-02-14 14:24:39,762 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36469.47 MB 2025-02-14 14:24:39,762 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:24:39,762 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33615.13 MB 2025-02-14 14:24:39,780 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8125, cut from 8127 2025-02-14 14:24:39,780 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 14:24:39,786 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 14:24:39,786 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 14:24:39,786 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:24:39,786 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:24:39,786 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23007.43 MB 2025-02-14 14:24:39,786 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31408.37 MB 2025-02-14 14:24:39,786 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8400.94 MB 2025-02-14 14:24:39,786 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36469.47 MB 2025-02-14 14:24:39,786 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44822.43 MB 2025-02-14 14:24:39,786 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8352.96 MB 2025-02-14 14:24:39,786 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31408.37 MB 2025-02-14 14:24:39,950 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7917] 2025-02-14 14:24:39,952 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:24:39,952 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 14:24:39,953 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:24:39,953 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 14:24:39,957 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 14:24:39,958 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:24:39,959 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 14:24:39,959 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 14:25:41,523 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:25:41,523 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 14:25:41,528 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 14:25:41,532 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:25:41,532 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1107, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 14:25:41,533 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:25:41,533 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1107, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 14:25:58,569 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 14:25:58,569 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 14:25:58,569 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.03 seconds 2025-02-14 14:25:58,569 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:25:58,569 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20682.46 MB 2025-02-14 14:25:58,569 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24600.07 MB 2025-02-14 14:25:58,569 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3917.61 MB 2025-02-14 14:25:58,569 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57350.82 MB 2025-02-14 14:25:58,569 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29183.97 MB 2025-02-14 14:25:58,569 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28166.85 MB 2025-02-14 14:25:58,569 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33552.02 MB 2025-02-14 14:25:58,636 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 14:25:58,636 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 14:25:58,636 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 14:25:58,636 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:25:58,636 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24600.07 MB 2025-02-14 14:25:58,636 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21532.80 MB 2025-02-14 14:25:58,636 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3067.27 MB 2025-02-14 14:25:58,636 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29183.97 MB 2025-02-14 14:25:58,636 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41217.43 MB 2025-02-14 14:25:58,636 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 12033.46 MB 2025-02-14 14:25:58,636 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35613.80 MB 2025-02-14 14:26:00,550 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 14:26:00,550 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 14:26:00,550 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 14:26:00,550 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:26:00,550 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21532.80 MB 2025-02-14 14:26:00,550 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22063.64 MB 2025-02-14 14:26:00,550 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 14:26:00,550 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41217.43 MB 2025-02-14 14:26:00,550 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26679.97 MB 2025-02-14 14:26:00,550 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14537.46 MB 2025-02-14 14:26:00,550 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26042.97 MB 2025-02-14 14:26:00,564 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 14:26:00,564 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 14:26:00,564 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 14:26:00,564 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:26:00,564 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22063.64 MB 2025-02-14 14:26:00,564 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23953.17 MB 2025-02-14 14:26:00,564 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 14:26:00,564 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26679.97 MB 2025-02-14 14:26:00,564 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28567.40 MB 2025-02-14 14:26:00,564 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 14:26:00,564 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25370.60 MB 2025-02-14 14:26:00,779 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 14:26:00,779 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 14:26:00,779 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 14:26:00,779 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:26:00,779 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23953.17 MB 2025-02-14 14:26:00,779 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26195.03 MB 2025-02-14 14:26:00,779 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 14:26:00,779 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28567.40 MB 2025-02-14 14:26:00,779 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34229.71 MB 2025-02-14 14:26:00,779 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 14:26:00,779 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31739.31 MB 2025-02-14 14:26:00,780 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 14:26:00,780 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 14:26:00,780 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 14:26:00,780 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:26:00,780 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22063.64 MB 2025-02-14 14:26:00,780 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26195.03 MB 2025-02-14 14:26:00,780 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 14:26:00,780 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26679.97 MB 2025-02-14 14:26:00,780 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34229.71 MB 2025-02-14 14:26:00,780 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-14 14:26:00,780 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31739.31 MB 2025-02-14 14:26:00,954 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 14:26:00,954 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 14:26:00,954 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 14:26:00,954 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:26:00,954 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27728.57 MB 2025-02-14 14:26:00,954 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28495.57 MB 2025-02-14 14:26:00,954 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 14:26:00,954 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34229.71 MB 2025-02-14 14:26:00,954 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34644.95 MB 2025-02-14 14:26:00,954 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 14:26:00,954 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29203.36 MB 2025-02-14 14:26:00,972 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 14:26:00,973 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 14:26:00,973 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:26:00,973 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:26:00,973 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28908.46 MB 2025-02-14 14:26:00,973 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29137.87 MB 2025-02-14 14:26:00,973 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.40 MB 2025-02-14 14:26:00,973 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34644.95 MB 2025-02-14 14:26:00,973 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34644.95 MB 2025-02-14 14:26:00,973 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:26:00,973 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29323.28 MB 2025-02-14 14:26:00,974 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 14:26:00,974 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 14:26:00,974 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.44 seconds 2025-02-14 14:26:00,974 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:26:00,974 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16825.58 MB 2025-02-14 14:26:00,974 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29338.94 MB 2025-02-14 14:26:00,974 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12513.36 MB 2025-02-14 14:26:00,974 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57350.82 MB 2025-02-14 14:26:00,974 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34644.95 MB 2025-02-14 14:26:00,974 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22705.86 MB 2025-02-14 14:26:00,974 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29338.94 MB 2025-02-14 14:26:01,243 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 14:26:01,243 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 14:26:01,243 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 14:26:01,243 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:26:01,243 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29338.94 MB 2025-02-14 14:26:01,243 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21829.97 MB 2025-02-14 14:26:01,243 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7508.97 MB 2025-02-14 14:26:01,243 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34644.95 MB 2025-02-14 14:26:01,243 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34644.95 MB 2025-02-14 14:26:01,243 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:26:01,243 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31850.61 MB 2025-02-14 14:26:01,261 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 14:26:01,261 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 14:26:01,267 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 14:26:01,267 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 14:26:01,267 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:26:01,267 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:26:01,267 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21829.97 MB 2025-02-14 14:26:01,267 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30269.00 MB 2025-02-14 14:26:01,267 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 14:26:01,267 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34644.95 MB 2025-02-14 14:26:01,268 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43035.66 MB 2025-02-14 14:26:01,268 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 14:26:01,268 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30269.00 MB 2025-02-14 14:26:01,437 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 14:26:01,438 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:26:01,438 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 14:26:01,439 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:26:01,439 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 14:26:01,444 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 14:26:01,445 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:26:01,445 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 14:26:01,445 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 14:26:07,519 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:26:07,519 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 14:26:07,526 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 14:26:07,529 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:26:07,529 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1484, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 14:26:07,530 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:26:07,530 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1484, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 14:26:30,675 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 14:26:30,675 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 14:26:30,675 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.14 seconds 2025-02-14 14:26:30,675 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:26:30,675 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23309.46 MB 2025-02-14 14:26:30,675 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28561.25 MB 2025-02-14 14:26:30,675 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5251.79 MB 2025-02-14 14:26:30,675 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55620.67 MB 2025-02-14 14:26:30,675 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38904.27 MB 2025-02-14 14:26:30,675 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16716.40 MB 2025-02-14 14:26:30,675 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37537.17 MB 2025-02-14 14:26:30,760 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 14:26:30,760 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 14:26:30,760 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 14:26:30,760 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:26:30,760 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28561.25 MB 2025-02-14 14:26:30,760 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23492.70 MB 2025-02-14 14:26:30,760 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5068.55 MB 2025-02-14 14:26:30,760 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38904.27 MB 2025-02-14 14:26:30,760 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49685.73 MB 2025-02-14 14:26:30,760 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10781.46 MB 2025-02-14 14:26:30,760 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44277.72 MB 2025-02-14 14:26:32,680 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 14:26:32,680 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 14:26:32,680 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 14:26:32,680 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:26:32,680 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23492.70 MB 2025-02-14 14:26:32,680 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24023.55 MB 2025-02-14 14:26:32,680 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 14:26:32,680 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49685.73 MB 2025-02-14 14:26:32,680 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29475.47 MB 2025-02-14 14:26:32,680 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20210.25 MB 2025-02-14 14:26:32,680 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28002.88 MB 2025-02-14 14:26:32,696 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 14:26:32,697 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 14:26:32,697 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 14:26:32,697 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:26:32,697 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24023.55 MB 2025-02-14 14:26:32,697 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25913.08 MB 2025-02-14 14:26:32,697 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 14:26:32,697 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29475.47 MB 2025-02-14 14:26:32,697 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30419.19 MB 2025-02-14 14:26:32,697 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 14:26:32,697 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27330.51 MB 2025-02-14 14:26:32,902 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 14:26:32,902 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 14:26:32,902 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 14:26:32,902 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:26:32,902 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25913.08 MB 2025-02-14 14:26:32,902 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28154.94 MB 2025-02-14 14:26:32,902 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 14:26:32,902 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30419.19 MB 2025-02-14 14:26:32,902 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36081.50 MB 2025-02-14 14:26:32,902 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 14:26:32,902 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33699.22 MB 2025-02-14 14:26:32,903 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 14:26:32,903 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 14:26:32,903 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 14:26:32,903 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:26:32,903 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24023.55 MB 2025-02-14 14:26:32,903 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28154.94 MB 2025-02-14 14:26:32,903 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 14:26:32,903 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29475.47 MB 2025-02-14 14:26:32,903 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36081.50 MB 2025-02-14 14:26:32,903 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 14:26:32,903 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33699.22 MB 2025-02-14 14:26:33,069 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 14:26:33,069 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 14:26:33,069 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 14:26:33,069 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:26:33,069 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29688.48 MB 2025-02-14 14:26:33,069 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30455.48 MB 2025-02-14 14:26:33,069 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 14:26:33,069 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36081.50 MB 2025-02-14 14:26:33,069 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36498.83 MB 2025-02-14 14:26:33,069 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 14:26:33,069 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31163.27 MB 2025-02-14 14:26:33,088 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 14:26:33,088 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 14:26:33,088 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:26:33,088 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:26:33,088 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30868.37 MB 2025-02-14 14:26:33,088 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31097.03 MB 2025-02-14 14:26:33,088 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.67 MB 2025-02-14 14:26:33,088 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36498.83 MB 2025-02-14 14:26:33,088 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36498.83 MB 2025-02-14 14:26:33,088 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:26:33,088 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31337.12 MB 2025-02-14 14:26:33,089 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 14:26:33,089 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 14:26:33,089 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.56 seconds 2025-02-14 14:26:33,089 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:26:33,089 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18139.08 MB 2025-02-14 14:26:33,089 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31297.62 MB 2025-02-14 14:26:33,089 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13158.53 MB 2025-02-14 14:26:33,089 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55620.67 MB 2025-02-14 14:26:33,089 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36498.83 MB 2025-02-14 14:26:33,089 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19121.83 MB 2025-02-14 14:26:33,089 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31337.12 MB 2025-02-14 14:26:33,357 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 14:26:33,358 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 14:26:33,358 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 14:26:33,358 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:26:33,358 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31297.62 MB 2025-02-14 14:26:33,358 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23135.85 MB 2025-02-14 14:26:33,358 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8161.76 MB 2025-02-14 14:26:33,358 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36498.83 MB 2025-02-14 14:26:33,358 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36498.83 MB 2025-02-14 14:26:33,358 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:26:33,358 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33803.14 MB 2025-02-14 14:26:33,375 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8142, cut from 8144 2025-02-14 14:26:33,376 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 14:26:33,382 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 14:26:33,382 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 14:26:33,382 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:26:33,382 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:26:33,382 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23135.85 MB 2025-02-14 14:26:33,382 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31554.01 MB 2025-02-14 14:26:33,382 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8418.15 MB 2025-02-14 14:26:33,382 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36498.83 MB 2025-02-14 14:26:33,382 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44868.57 MB 2025-02-14 14:26:33,382 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8369.73 MB 2025-02-14 14:26:33,382 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31554.01 MB 2025-02-14 14:26:33,537 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7934] 2025-02-14 14:26:33,539 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:26:33,539 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 14:26:33,540 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:26:33,540 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 14:26:33,544 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 14:26:33,545 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:26:33,545 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 14:26:33,545 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 14:27:32,273 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:27:32,273 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 14:27:32,278 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 14:27:32,282 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:27:32,282 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 78, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 14:27:32,283 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:27:32,283 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 78, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 14:27:33,520 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 14:27:33,520 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 14:27:33,520 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.23 seconds 2025-02-14 14:27:33,520 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:27:33,520 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13512.22 MB 2025-02-14 14:27:33,520 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 13788.26 MB 2025-02-14 14:27:33,520 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 276.04 MB 2025-02-14 14:27:33,520 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57422.12 MB 2025-02-14 14:27:33,520 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22032.68 MB 2025-02-14 14:27:33,520 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35389.44 MB 2025-02-14 14:27:33,520 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22757.10 MB 2025-02-14 14:27:33,523 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 14:27:33,523 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 14:27:33,523 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 14:27:33,523 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:27:33,523 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13788.26 MB 2025-02-14 14:27:33,523 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 13922.00 MB 2025-02-14 14:27:33,523 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 133.74 MB 2025-02-14 14:27:33,523 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22032.68 MB 2025-02-14 14:27:33,523 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22032.68 MB 2025-02-14 14:27:33,523 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:27:33,523 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14336.12 MB 2025-02-14 14:27:33,911 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 14:27:33,911 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 14:27:33,911 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.39 seconds 2025-02-14 14:27:33,911 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:27:33,911 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13922.00 MB 2025-02-14 14:27:33,911 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14025.51 MB 2025-02-14 14:27:33,911 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 103.51 MB 2025-02-14 14:27:33,911 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22032.68 MB 2025-02-14 14:27:33,911 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22032.68 MB 2025-02-14 14:27:33,911 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:27:33,911 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18007.50 MB 2025-02-14 14:27:33,917 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 14:27:33,917 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 14:27:33,917 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 14:27:33,917 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:27:33,917 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14025.45 MB 2025-02-14 14:27:33,917 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14393.82 MB 2025-02-14 14:27:33,917 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 368.37 MB 2025-02-14 14:27:33,917 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22032.68 MB 2025-02-14 14:27:33,917 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22032.68 MB 2025-02-14 14:27:33,917 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:27:33,917 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 14670.22 MB 2025-02-14 14:27:33,995 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 14:27:33,996 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 14:27:33,996 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 14:27:33,996 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:27:33,996 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14393.82 MB 2025-02-14 14:27:33,996 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14841.26 MB 2025-02-14 14:27:33,996 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 447.44 MB 2025-02-14 14:27:33,996 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22032.68 MB 2025-02-14 14:27:33,996 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22032.68 MB 2025-02-14 14:27:33,996 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:27:33,996 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15912.12 MB 2025-02-14 14:27:33,996 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 14:27:33,996 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 14:27:33,996 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 14:27:33,996 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:27:33,996 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14025.45 MB 2025-02-14 14:27:33,996 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14841.26 MB 2025-02-14 14:27:33,996 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 815.81 MB 2025-02-14 14:27:33,996 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22032.68 MB 2025-02-14 14:27:33,996 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22032.68 MB 2025-02-14 14:27:33,996 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:27:33,997 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15912.12 MB 2025-02-14 14:27:34,040 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 14:27:34,040 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 14:27:34,040 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-14 14:27:34,040 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:27:34,040 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15273.20 MB 2025-02-14 14:27:34,040 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15461.11 MB 2025-02-14 14:27:34,040 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 187.90 MB 2025-02-14 14:27:34,040 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22032.68 MB 2025-02-14 14:27:34,040 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22150.12 MB 2025-02-14 14:27:34,040 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 117.44 MB 2025-02-14 14:27:34,040 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15599.13 MB 2025-02-14 14:27:34,045 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 14:27:34,045 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 14:27:34,045 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 14:27:34,045 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:27:34,045 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15579.97 MB 2025-02-14 14:27:34,045 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15767.33 MB 2025-02-14 14:27:34,045 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 187.36 MB 2025-02-14 14:27:34,045 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22150.12 MB 2025-02-14 14:27:34,045 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22150.12 MB 2025-02-14 14:27:34,045 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:27:34,046 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15767.33 MB 2025-02-14 14:27:34,047 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 14:27:34,047 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 14:27:34,047 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.76 seconds 2025-02-14 14:27:34,047 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:27:34,047 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13240.46 MB 2025-02-14 14:27:34,047 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15934.29 MB 2025-02-14 14:27:34,047 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2693.83 MB 2025-02-14 14:27:34,047 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57422.12 MB 2025-02-14 14:27:34,047 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22150.12 MB 2025-02-14 14:27:34,047 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35272.00 MB 2025-02-14 14:27:34,047 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15934.29 MB 2025-02-14 14:27:34,266 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 14:27:34,266 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 14:27:34,266 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 14:27:34,266 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:27:34,266 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15934.29 MB 2025-02-14 14:27:34,266 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16196.85 MB 2025-02-14 14:27:34,266 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 262.56 MB 2025-02-14 14:27:34,266 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22150.12 MB 2025-02-14 14:27:34,266 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22150.12 MB 2025-02-14 14:27:34,266 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:27:34,266 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16447.10 MB 2025-02-14 14:27:34,281 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 6775, cut from 6777 2025-02-14 14:27:34,282 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 14:27:34,287 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 14:27:34,287 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 14:27:34,287 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:27:34,287 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:27:34,287 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16196.85 MB 2025-02-14 14:27:34,287 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23204.34 MB 2025-02-14 14:27:34,287 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7007.49 MB 2025-02-14 14:27:34,287 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22150.12 MB 2025-02-14 14:27:34,287 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25633.49 MB 2025-02-14 14:27:34,287 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3483.37 MB 2025-02-14 14:27:34,287 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23204.34 MB 2025-02-14 14:27:34,416 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 6567] 2025-02-14 14:27:34,417 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:27:34,417 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 14:27:34,418 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:27:34,418 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 14:27:34,423 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 14:27:34,425 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:27:34,425 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 14:27:34,425 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 14:28:19,145 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:28:19,145 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 14:28:19,150 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 14:28:19,153 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:28:19,154 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1265, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 14:28:19,154 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:28:19,155 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1265, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 14:28:38,596 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 14:28:38,597 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 14:28:38,597 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.43 seconds 2025-02-14 14:28:38,597 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:28:38,597 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21783.43 MB 2025-02-14 14:28:38,597 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26260.85 MB 2025-02-14 14:28:38,597 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4477.42 MB 2025-02-14 14:28:38,597 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32600.23 MB 2025-02-14 14:28:38,597 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36016.49 MB 2025-02-14 14:28:38,597 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3416.26 MB 2025-02-14 14:28:38,597 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35105.17 MB 2025-02-14 14:28:38,676 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 14:28:38,676 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 14:28:38,676 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 14:28:38,676 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:28:38,676 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26260.85 MB 2025-02-14 14:28:38,676 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22354.19 MB 2025-02-14 14:28:38,676 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3906.66 MB 2025-02-14 14:28:38,676 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36016.49 MB 2025-02-14 14:28:38,676 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46152.02 MB 2025-02-14 14:28:38,676 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10135.54 MB 2025-02-14 14:28:38,676 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39374.14 MB 2025-02-14 14:28:40,592 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 14:28:40,592 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 14:28:40,592 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 14:28:40,592 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:28:40,592 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22354.19 MB 2025-02-14 14:28:40,592 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22885.03 MB 2025-02-14 14:28:40,592 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 14:28:40,592 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46152.02 MB 2025-02-14 14:28:40,592 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29471.28 MB 2025-02-14 14:28:40,592 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16680.75 MB 2025-02-14 14:28:40,592 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26864.37 MB 2025-02-14 14:28:40,605 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 14:28:40,605 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 14:28:40,605 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 14:28:40,605 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:28:40,605 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22885.03 MB 2025-02-14 14:28:40,605 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24774.57 MB 2025-02-14 14:28:40,605 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 14:28:40,605 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29471.28 MB 2025-02-14 14:28:40,605 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29471.28 MB 2025-02-14 14:28:40,606 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:28:40,606 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26192.00 MB 2025-02-14 14:28:40,818 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 14:28:40,818 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 14:28:40,818 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 14:28:40,818 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:28:40,818 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24774.57 MB 2025-02-14 14:28:40,818 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27016.42 MB 2025-02-14 14:28:40,818 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 14:28:40,818 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29471.28 MB 2025-02-14 14:28:40,818 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35133.59 MB 2025-02-14 14:28:40,818 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 14:28:40,818 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32560.70 MB 2025-02-14 14:28:40,819 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 14:28:40,819 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 14:28:40,819 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 14:28:40,819 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:28:40,819 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22885.03 MB 2025-02-14 14:28:40,819 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27016.42 MB 2025-02-14 14:28:40,819 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 14:28:40,819 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29471.28 MB 2025-02-14 14:28:40,819 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35133.59 MB 2025-02-14 14:28:40,819 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 14:28:40,819 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32560.70 MB 2025-02-14 14:28:41,003 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 14:28:41,003 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 14:28:41,003 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-14 14:28:41,003 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:28:41,003 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28549.97 MB 2025-02-14 14:28:41,003 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29316.97 MB 2025-02-14 14:28:41,003 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 14:28:41,003 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35133.59 MB 2025-02-14 14:28:41,003 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35548.82 MB 2025-02-14 14:28:41,003 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 14:28:41,003 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30024.76 MB 2025-02-14 14:28:41,022 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 14:28:41,022 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 14:28:41,022 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:28:41,022 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:28:41,022 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29729.86 MB 2025-02-14 14:28:41,022 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29958.69 MB 2025-02-14 14:28:41,022 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.84 MB 2025-02-14 14:28:41,022 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35548.82 MB 2025-02-14 14:28:41,022 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35548.82 MB 2025-02-14 14:28:41,022 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:28:41,022 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30173.73 MB 2025-02-14 14:28:41,023 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 14:28:41,023 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 14:28:41,023 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.87 seconds 2025-02-14 14:28:41,023 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:28:41,023 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17376.07 MB 2025-02-14 14:28:41,023 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30159.45 MB 2025-02-14 14:28:41,023 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12783.38 MB 2025-02-14 14:28:41,023 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32600.23 MB 2025-02-14 14:28:41,023 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35548.82 MB 2025-02-14 14:28:41,023 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2948.60 MB 2025-02-14 14:28:41,023 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30173.73 MB 2025-02-14 14:28:41,293 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 14:28:41,293 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 14:28:41,293 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 14:28:41,293 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:28:41,293 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30159.45 MB 2025-02-14 14:28:41,293 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22375.50 MB 2025-02-14 14:28:41,293 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7783.94 MB 2025-02-14 14:28:41,293 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35548.82 MB 2025-02-14 14:28:41,293 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35548.82 MB 2025-02-14 14:28:41,293 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:28:41,293 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32667.12 MB 2025-02-14 14:28:41,310 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8149, cut from 8151 2025-02-14 14:28:41,311 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 14:28:41,317 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 14:28:41,317 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 14:28:41,317 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:28:41,317 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:28:41,317 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22375.50 MB 2025-02-14 14:28:41,317 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30801.68 MB 2025-02-14 14:28:41,317 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8426.18 MB 2025-02-14 14:28:41,317 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35548.82 MB 2025-02-14 14:28:41,317 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43924.85 MB 2025-02-14 14:28:41,317 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8376.03 MB 2025-02-14 14:28:41,317 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30801.68 MB 2025-02-14 14:28:41,478 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7941] 2025-02-14 14:28:41,480 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:28:41,480 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 14:28:41,481 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:28:41,481 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 14:28:41,486 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 14:28:41,487 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:28:41,487 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 14:28:41,487 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 14:28:54,212 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:28:54,212 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 14:28:54,217 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 14:28:54,220 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:28:54,221 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1074, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 14:28:54,222 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:28:54,222 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1074, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 14:29:10,897 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 14:29:10,897 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 14:29:10,897 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.67 seconds 2025-02-14 14:29:10,897 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:29:10,897 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20452.51 MB 2025-02-14 14:29:10,897 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24253.34 MB 2025-02-14 14:29:10,897 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3800.83 MB 2025-02-14 14:29:10,897 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52300.87 MB 2025-02-14 14:29:10,897 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31859.93 MB 2025-02-14 14:29:10,897 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20440.94 MB 2025-02-14 14:29:10,897 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33094.78 MB 2025-02-14 14:29:10,956 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 14:29:10,956 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 14:29:10,956 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 14:29:10,956 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:29:10,956 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24253.34 MB 2025-02-14 14:29:10,957 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21361.24 MB 2025-02-14 14:29:10,957 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2892.09 MB 2025-02-14 14:29:10,957 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31859.93 MB 2025-02-14 14:29:10,957 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37354.47 MB 2025-02-14 14:29:10,957 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5494.54 MB 2025-02-14 14:29:10,957 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33936.12 MB 2025-02-14 14:29:12,889 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 14:29:12,889 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 14:29:12,889 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 14:29:12,889 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:29:12,889 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21361.24 MB 2025-02-14 14:29:12,889 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21892.08 MB 2025-02-14 14:29:12,889 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 14:29:12,889 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37354.47 MB 2025-02-14 14:29:12,889 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25990.00 MB 2025-02-14 14:29:12,889 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11364.47 MB 2025-02-14 14:29:12,889 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25871.42 MB 2025-02-14 14:29:12,903 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 14:29:12,903 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 14:29:12,903 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 14:29:12,903 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:29:12,903 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21892.08 MB 2025-02-14 14:29:12,903 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23781.62 MB 2025-02-14 14:29:12,903 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 14:29:12,903 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25990.00 MB 2025-02-14 14:29:12,903 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27877.44 MB 2025-02-14 14:29:12,904 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 14:29:12,904 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25199.05 MB 2025-02-14 14:29:13,115 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 14:29:13,115 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 14:29:13,115 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 14:29:13,115 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:29:13,115 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23781.62 MB 2025-02-14 14:29:13,115 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26023.47 MB 2025-02-14 14:29:13,115 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 14:29:13,115 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27877.44 MB 2025-02-14 14:29:13,115 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34011.61 MB 2025-02-14 14:29:13,115 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 14:29:13,115 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31567.76 MB 2025-02-14 14:29:13,116 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 14:29:13,116 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 14:29:13,116 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 14:29:13,116 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:29:13,116 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21892.08 MB 2025-02-14 14:29:13,116 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26023.47 MB 2025-02-14 14:29:13,116 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 14:29:13,116 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25990.00 MB 2025-02-14 14:29:13,116 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34011.61 MB 2025-02-14 14:29:13,116 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-14 14:29:13,116 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31567.76 MB 2025-02-14 14:29:13,289 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 14:29:13,289 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 14:29:13,289 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 14:29:13,289 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:29:13,289 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27557.02 MB 2025-02-14 14:29:13,289 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28324.02 MB 2025-02-14 14:29:13,289 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 14:29:13,289 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34011.61 MB 2025-02-14 14:29:13,289 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34424.75 MB 2025-02-14 14:29:13,289 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 14:29:13,289 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29031.81 MB 2025-02-14 14:29:13,308 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 14:29:13,308 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 14:29:13,308 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:29:13,308 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:29:13,308 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28736.91 MB 2025-02-14 14:29:13,308 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28966.29 MB 2025-02-14 14:29:13,308 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.38 MB 2025-02-14 14:29:13,308 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34424.75 MB 2025-02-14 14:29:13,308 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34424.75 MB 2025-02-14 14:29:13,308 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:29:13,308 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29183.81 MB 2025-02-14 14:29:13,309 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 14:29:13,309 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 14:29:13,309 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.09 seconds 2025-02-14 14:29:13,309 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:29:13,309 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16710.61 MB 2025-02-14 14:29:13,309 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29167.36 MB 2025-02-14 14:29:13,309 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12456.75 MB 2025-02-14 14:29:13,309 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52300.87 MB 2025-02-14 14:29:13,309 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34424.75 MB 2025-02-14 14:29:13,309 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17876.12 MB 2025-02-14 14:29:13,309 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29183.81 MB 2025-02-14 14:29:13,580 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 14:29:13,580 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 14:29:13,580 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 14:29:13,580 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:29:13,580 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29167.36 MB 2025-02-14 14:29:13,580 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21715.00 MB 2025-02-14 14:29:13,580 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7452.36 MB 2025-02-14 14:29:13,580 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34424.75 MB 2025-02-14 14:29:13,580 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34424.75 MB 2025-02-14 14:29:13,580 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:29:13,580 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31679.03 MB 2025-02-14 14:29:13,598 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 14:29:13,598 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 14:29:13,604 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 14:29:13,604 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 14:29:13,604 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:29:13,604 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:29:13,604 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21715.00 MB 2025-02-14 14:29:13,604 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30154.02 MB 2025-02-14 14:29:13,604 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 14:29:13,604 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34424.75 MB 2025-02-14 14:29:13,604 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42815.46 MB 2025-02-14 14:29:13,604 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 14:29:13,604 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30154.02 MB 2025-02-14 14:29:13,768 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 14:29:13,770 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:29:13,770 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 14:29:13,770 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:29:13,771 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 14:29:13,775 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 14:29:13,776 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:29:13,776 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 14:29:13,776 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 14:30:22,216 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:30:22,216 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 14:30:22,221 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 14:30:22,225 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:30:22,225 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 211, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 14:30:22,226 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:30:22,226 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 211, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 14:30:25,491 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 14:30:25,491 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 14:30:25,491 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.26 seconds 2025-02-14 14:30:25,491 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:30:25,491 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14438.99 MB 2025-02-14 14:30:25,491 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15185.71 MB 2025-02-14 14:30:25,491 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 746.72 MB 2025-02-14 14:30:25,491 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55400.46 MB 2025-02-14 14:30:25,491 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18327.01 MB 2025-02-14 14:30:25,491 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -37073.45 MB 2025-02-14 14:30:25,491 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24137.66 MB 2025-02-14 14:30:25,505 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 14:30:25,505 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 14:30:25,505 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 14:30:25,505 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:30:25,505 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15185.71 MB 2025-02-14 14:30:25,505 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15267.49 MB 2025-02-14 14:30:25,505 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 81.78 MB 2025-02-14 14:30:25,505 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18327.01 MB 2025-02-14 14:30:25,505 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18933.09 MB 2025-02-14 14:30:25,505 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 606.08 MB 2025-02-14 14:30:25,505 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17599.18 MB 2025-02-14 14:30:26,321 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 14:30:26,321 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 14:30:26,321 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.81 seconds 2025-02-14 14:30:26,321 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:30:26,321 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15267.49 MB 2025-02-14 14:30:26,321 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15494.42 MB 2025-02-14 14:30:26,321 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.93 MB 2025-02-14 14:30:26,321 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18933.09 MB 2025-02-14 14:30:26,321 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18933.09 MB 2025-02-14 14:30:26,321 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:30:26,321 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19438.96 MB 2025-02-14 14:30:26,330 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 14:30:26,330 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 14:30:26,330 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 14:30:26,330 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:30:26,330 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15494.35 MB 2025-02-14 14:30:26,330 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16301.93 MB 2025-02-14 14:30:26,330 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 807.58 MB 2025-02-14 14:30:26,330 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18933.09 MB 2025-02-14 14:30:26,330 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18933.09 MB 2025-02-14 14:30:26,330 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:30:26,330 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16907.89 MB 2025-02-14 14:30:26,427 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 14:30:26,427 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 14:30:26,427 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 14:30:26,427 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:30:26,427 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16301.93 MB 2025-02-14 14:30:26,427 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17260.36 MB 2025-02-14 14:30:26,427 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 958.43 MB 2025-02-14 14:30:26,427 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18933.09 MB 2025-02-14 14:30:26,427 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20956.84 MB 2025-02-14 14:30:26,427 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2023.75 MB 2025-02-14 14:30:26,427 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19630.51 MB 2025-02-14 14:30:26,428 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 14:30:26,428 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 14:30:26,428 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 14:30:26,428 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:30:26,428 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15494.35 MB 2025-02-14 14:30:26,428 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17260.36 MB 2025-02-14 14:30:26,428 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1766.01 MB 2025-02-14 14:30:26,428 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18933.09 MB 2025-02-14 14:30:26,428 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20956.84 MB 2025-02-14 14:30:26,428 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2023.75 MB 2025-02-14 14:30:26,428 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19630.51 MB 2025-02-14 14:30:26,503 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 14:30:26,503 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 14:30:26,503 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 14:30:26,503 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:30:26,503 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17915.95 MB 2025-02-14 14:30:26,503 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18243.85 MB 2025-02-14 14:30:26,503 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 327.89 MB 2025-02-14 14:30:26,503 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20956.84 MB 2025-02-14 14:30:26,503 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21130.90 MB 2025-02-14 14:30:26,503 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 174.06 MB 2025-02-14 14:30:26,503 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18553.57 MB 2025-02-14 14:30:26,513 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 14:30:26,513 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 14:30:26,513 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 14:30:26,513 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:30:26,513 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18420.37 MB 2025-02-14 14:30:26,513 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18629.50 MB 2025-02-14 14:30:26,513 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 209.13 MB 2025-02-14 14:30:26,513 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21130.90 MB 2025-02-14 14:30:26,513 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21130.90 MB 2025-02-14 14:30:26,513 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:30:26,513 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18654.33 MB 2025-02-14 14:30:26,514 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 14:30:26,514 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 14:30:26,514 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.29 seconds 2025-02-14 14:30:26,514 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:30:26,514 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13703.85 MB 2025-02-14 14:30:26,514 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18830.32 MB 2025-02-14 14:30:26,514 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5126.48 MB 2025-02-14 14:30:26,514 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55400.46 MB 2025-02-14 14:30:26,514 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21130.90 MB 2025-02-14 14:30:26,514 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34269.56 MB 2025-02-14 14:30:26,515 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18830.32 MB 2025-02-14 14:30:26,780 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 14:30:26,780 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 14:30:26,780 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 14:30:26,780 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:30:26,780 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18830.32 MB 2025-02-14 14:30:26,780 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17623.71 MB 2025-02-14 14:30:26,780 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1206.61 MB 2025-02-14 14:30:26,780 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21130.90 MB 2025-02-14 14:30:26,780 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21130.90 MB 2025-02-14 14:30:26,780 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:30:26,780 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19064.46 MB 2025-02-14 14:30:26,798 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8152, cut from 8154 2025-02-14 14:30:26,798 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 14:30:26,804 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 14:30:26,804 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 14:30:26,804 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:30:26,804 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:30:26,804 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17623.71 MB 2025-02-14 14:30:26,804 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26052.83 MB 2025-02-14 14:30:26,804 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8429.12 MB 2025-02-14 14:30:26,804 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21130.90 MB 2025-02-14 14:30:26,804 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31606.18 MB 2025-02-14 14:30:26,804 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10475.27 MB 2025-02-14 14:30:26,804 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26052.83 MB 2025-02-14 14:30:26,961 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7944] 2025-02-14 14:30:26,963 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:30:26,963 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 14:30:26,964 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:30:26,964 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 14:30:26,968 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 14:30:26,969 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:30:26,969 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 14:30:26,970 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 14:31:18,676 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:31:18,676 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 14:31:18,681 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 14:31:18,685 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:31:18,685 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1655, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 14:31:18,686 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:31:18,686 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1655, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 14:31:44,167 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 14:31:44,168 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 14:31:44,168 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.47 seconds 2025-02-14 14:31:44,168 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:31:44,168 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24501.01 MB 2025-02-14 14:31:44,168 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30358.36 MB 2025-02-14 14:31:44,168 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5857.35 MB 2025-02-14 14:31:44,168 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39986.40 MB 2025-02-14 14:31:44,168 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39527.12 MB 2025-02-14 14:31:44,168 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -459.28 MB 2025-02-14 14:31:44,168 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39182.51 MB 2025-02-14 14:31:44,271 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 14:31:44,271 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 14:31:44,271 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 14:31:44,271 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:31:44,271 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30358.36 MB 2025-02-14 14:31:44,271 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24381.68 MB 2025-02-14 14:31:44,271 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5976.68 MB 2025-02-14 14:31:44,271 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39527.12 MB 2025-02-14 14:31:44,271 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56568.58 MB 2025-02-14 14:31:44,271 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 17041.46 MB 2025-02-14 14:31:44,271 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47719.26 MB 2025-02-14 14:31:46,199 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 14:31:46,199 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 14:31:46,199 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 14:31:46,199 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:31:46,199 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24381.68 MB 2025-02-14 14:31:46,199 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24912.52 MB 2025-02-14 14:31:46,199 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 14:31:46,199 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56568.58 MB 2025-02-14 14:31:46,199 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30895.24 MB 2025-02-14 14:31:46,199 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25673.33 MB 2025-02-14 14:31:46,199 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28891.97 MB 2025-02-14 14:31:46,215 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 14:31:46,216 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 14:31:46,216 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 14:31:46,216 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:31:46,216 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24912.52 MB 2025-02-14 14:31:46,216 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26802.06 MB 2025-02-14 14:31:46,216 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 14:31:46,216 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30895.24 MB 2025-02-14 14:31:46,216 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31838.96 MB 2025-02-14 14:31:46,216 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 14:31:46,216 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28219.48 MB 2025-02-14 14:31:46,430 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 14:31:46,430 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 14:31:46,430 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 14:31:46,430 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:31:46,430 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26802.06 MB 2025-02-14 14:31:46,430 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29043.91 MB 2025-02-14 14:31:46,430 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 14:31:46,430 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31838.96 MB 2025-02-14 14:31:46,430 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37501.27 MB 2025-02-14 14:31:46,430 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 14:31:46,430 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34588.19 MB 2025-02-14 14:31:46,431 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 14:31:46,431 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 14:31:46,431 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 14:31:46,431 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:31:46,431 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24912.52 MB 2025-02-14 14:31:46,431 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29043.91 MB 2025-02-14 14:31:46,431 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 14:31:46,431 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30895.24 MB 2025-02-14 14:31:46,431 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37501.27 MB 2025-02-14 14:31:46,431 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 14:31:46,431 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34588.19 MB 2025-02-14 14:31:46,601 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 14:31:46,601 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 14:31:46,601 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 14:31:46,601 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:31:46,601 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30577.45 MB 2025-02-14 14:31:46,601 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31344.46 MB 2025-02-14 14:31:46,601 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 14:31:46,601 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37501.27 MB 2025-02-14 14:31:46,601 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37916.51 MB 2025-02-14 14:31:46,601 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 14:31:46,601 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32052.24 MB 2025-02-14 14:31:46,620 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 14:31:46,620 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 14:31:46,620 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:31:46,620 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:31:46,620 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31757.34 MB 2025-02-14 14:31:46,621 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31985.40 MB 2025-02-14 14:31:46,621 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.05 MB 2025-02-14 14:31:46,621 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37916.51 MB 2025-02-14 14:31:46,621 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37916.51 MB 2025-02-14 14:31:46,621 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:31:46,621 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32229.18 MB 2025-02-14 14:31:46,622 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 14:31:46,622 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 14:31:46,622 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.93 seconds 2025-02-14 14:31:46,622 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:31:46,622 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18734.86 MB 2025-02-14 14:31:46,622 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32185.36 MB 2025-02-14 14:31:46,622 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13450.50 MB 2025-02-14 14:31:46,622 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39986.40 MB 2025-02-14 14:31:46,622 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37916.51 MB 2025-02-14 14:31:46,622 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2069.89 MB 2025-02-14 14:31:46,622 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32229.18 MB 2025-02-14 14:31:46,890 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 14:31:46,890 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 14:31:46,890 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 14:31:46,890 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:31:46,890 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32185.36 MB 2025-02-14 14:31:46,890 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23722.11 MB 2025-02-14 14:31:46,890 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8463.26 MB 2025-02-14 14:31:46,890 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37916.51 MB 2025-02-14 14:31:46,890 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37916.51 MB 2025-02-14 14:31:46,890 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:31:46,890 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34683.21 MB 2025-02-14 14:31:46,908 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8117, cut from 8119 2025-02-14 14:31:46,908 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 14:31:46,914 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 14:31:46,914 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 14:31:46,914 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:31:46,914 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:31:46,914 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23722.11 MB 2025-02-14 14:31:46,914 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32114.70 MB 2025-02-14 14:31:46,914 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8392.59 MB 2025-02-14 14:31:46,914 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37916.51 MB 2025-02-14 14:31:46,914 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46261.08 MB 2025-02-14 14:31:46,914 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8344.57 MB 2025-02-14 14:31:46,914 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32114.70 MB 2025-02-14 14:31:47,076 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7909] 2025-02-14 14:31:47,077 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:31:47,077 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 14:31:47,078 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:31:47,078 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 14:31:47,083 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 14:31:47,084 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:31:47,084 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 14:31:47,084 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 14:31:55,099 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:31:55,099 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 14:31:55,105 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 14:31:55,109 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:31:55,109 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1241, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 14:31:55,110 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:31:55,110 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1241, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 14:32:14,441 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 14:32:14,441 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 14:32:14,441 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.33 seconds 2025-02-14 14:32:14,441 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:32:14,441 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21616.19 MB 2025-02-14 14:32:14,441 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26008.02 MB 2025-02-14 14:32:14,441 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4391.83 MB 2025-02-14 14:32:14,441 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58776.88 MB 2025-02-14 14:32:14,441 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37987.81 MB 2025-02-14 14:32:14,441 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20789.07 MB 2025-02-14 14:32:14,441 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34937.94 MB 2025-02-14 14:32:14,513 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 14:32:14,513 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 14:32:14,513 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 14:32:14,513 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:32:14,513 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26008.02 MB 2025-02-14 14:32:14,513 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22229.42 MB 2025-02-14 14:32:14,513 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3778.60 MB 2025-02-14 14:32:14,513 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37987.81 MB 2025-02-14 14:32:14,513 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46558.87 MB 2025-02-14 14:32:14,513 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8571.06 MB 2025-02-14 14:32:14,513 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38927.95 MB 2025-02-14 14:32:16,448 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 14:32:16,448 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 14:32:16,448 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 14:32:16,448 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:32:16,448 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22229.42 MB 2025-02-14 14:32:16,448 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22760.26 MB 2025-02-14 14:32:16,448 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 14:32:16,448 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46558.87 MB 2025-02-14 14:32:16,448 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33594.28 MB 2025-02-14 14:32:16,448 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12964.59 MB 2025-02-14 14:32:16,448 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26739.60 MB 2025-02-14 14:32:16,461 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 14:32:16,461 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 14:32:16,461 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 14:32:16,461 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:32:16,461 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22760.26 MB 2025-02-14 14:32:16,461 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24649.80 MB 2025-02-14 14:32:16,461 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 14:32:16,461 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33594.28 MB 2025-02-14 14:32:16,461 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33594.28 MB 2025-02-14 14:32:16,461 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:32:16,461 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26067.23 MB 2025-02-14 14:32:16,697 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 14:32:16,697 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 14:32:16,697 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 14:32:16,697 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:32:16,697 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24649.80 MB 2025-02-14 14:32:16,697 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26891.65 MB 2025-02-14 14:32:16,697 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 14:32:16,697 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33594.28 MB 2025-02-14 14:32:16,697 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35481.71 MB 2025-02-14 14:32:16,697 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 14:32:16,697 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32435.94 MB 2025-02-14 14:32:16,698 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 14:32:16,698 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 14:32:16,698 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.25 seconds 2025-02-14 14:32:16,698 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:32:16,698 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22760.26 MB 2025-02-14 14:32:16,698 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26891.65 MB 2025-02-14 14:32:16,698 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 14:32:16,698 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33594.28 MB 2025-02-14 14:32:16,698 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35481.71 MB 2025-02-14 14:32:16,698 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 14:32:16,698 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32435.94 MB 2025-02-14 14:32:16,867 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 14:32:16,867 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 14:32:16,867 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 14:32:16,867 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:32:16,867 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28425.20 MB 2025-02-14 14:32:16,867 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29192.20 MB 2025-02-14 14:32:16,867 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 14:32:16,867 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35481.71 MB 2025-02-14 14:32:16,867 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35899.05 MB 2025-02-14 14:32:16,867 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 14:32:16,867 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29899.99 MB 2025-02-14 14:32:16,886 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 14:32:16,886 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 14:32:16,886 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:32:16,886 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:32:16,886 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29605.09 MB 2025-02-14 14:32:16,886 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29832.43 MB 2025-02-14 14:32:16,886 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.34 MB 2025-02-14 14:32:16,886 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35899.05 MB 2025-02-14 14:32:16,886 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35899.05 MB 2025-02-14 14:32:16,886 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:32:16,886 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30068.12 MB 2025-02-14 14:32:16,887 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 14:32:16,887 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 14:32:16,887 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.78 seconds 2025-02-14 14:32:16,887 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:32:16,887 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17292.45 MB 2025-02-14 14:32:16,887 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30033.30 MB 2025-02-14 14:32:16,887 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12740.85 MB 2025-02-14 14:32:16,887 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58776.88 MB 2025-02-14 14:32:16,887 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35899.05 MB 2025-02-14 14:32:16,887 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22877.83 MB 2025-02-14 14:32:16,887 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30068.12 MB 2025-02-14 14:32:17,156 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 14:32:17,156 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 14:32:17,156 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 14:32:17,156 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:32:17,156 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30033.30 MB 2025-02-14 14:32:17,156 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22293.79 MB 2025-02-14 14:32:17,156 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7739.51 MB 2025-02-14 14:32:17,156 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35899.05 MB 2025-02-14 14:32:17,156 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35899.05 MB 2025-02-14 14:32:17,156 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:32:17,157 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32542.51 MB 2025-02-14 14:32:17,174 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8154, cut from 8156 2025-02-14 14:32:17,175 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 14:32:17,181 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 14:32:17,181 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 14:32:17,181 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:32:17,181 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:32:17,181 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22293.79 MB 2025-02-14 14:32:17,181 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30724.47 MB 2025-02-14 14:32:17,181 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8430.68 MB 2025-02-14 14:32:17,181 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35899.05 MB 2025-02-14 14:32:17,181 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44281.36 MB 2025-02-14 14:32:17,181 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8382.32 MB 2025-02-14 14:32:17,181 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30724.47 MB 2025-02-14 14:32:17,343 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7946] 2025-02-14 14:32:17,345 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:32:17,345 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 14:32:17,346 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:32:17,346 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 14:32:17,350 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 14:32:17,351 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:32:17,351 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 14:32:17,352 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 14:33:21,161 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:33:21,161 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 14:33:21,167 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 14:33:21,171 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:33:21,171 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 128, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 14:33:21,172 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:33:21,172 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 128, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 14:33:23,160 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 14:33:23,160 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 14:33:23,160 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.98 seconds 2025-02-14 14:33:23,160 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:33:23,160 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13860.63 MB 2025-02-14 14:33:23,160 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14313.62 MB 2025-02-14 14:33:23,160 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 452.98 MB 2025-02-14 14:33:23,160 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56853.79 MB 2025-02-14 14:33:23,160 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17379.10 MB 2025-02-14 14:33:23,160 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -39474.69 MB 2025-02-14 14:33:23,160 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23332.81 MB 2025-02-14 14:33:23,170 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 14:33:23,170 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 14:33:23,170 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 14:33:23,170 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:33:23,170 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14313.62 MB 2025-02-14 14:33:23,170 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14533.09 MB 2025-02-14 14:33:23,170 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 219.47 MB 2025-02-14 14:33:23,170 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17379.10 MB 2025-02-14 14:33:23,170 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17379.10 MB 2025-02-14 14:33:23,170 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:33:23,170 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16168.98 MB 2025-02-14 14:33:23,781 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 14:33:23,781 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 14:33:23,781 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.61 seconds 2025-02-14 14:33:23,781 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:33:23,781 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14533.09 MB 2025-02-14 14:33:23,781 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14702.95 MB 2025-02-14 14:33:23,781 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 169.87 MB 2025-02-14 14:33:23,781 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17379.10 MB 2025-02-14 14:33:23,781 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17379.10 MB 2025-02-14 14:33:23,781 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:33:23,781 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18704.56 MB 2025-02-14 14:33:23,788 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 14:33:23,788 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 14:33:23,788 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 14:33:23,788 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:33:23,788 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14702.89 MB 2025-02-14 14:33:23,788 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15307.39 MB 2025-02-14 14:33:23,788 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 604.50 MB 2025-02-14 14:33:23,788 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17379.10 MB 2025-02-14 14:33:23,788 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17379.10 MB 2025-02-14 14:33:23,788 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:33:23,788 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15760.98 MB 2025-02-14 14:33:23,859 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 14:33:23,859 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 14:33:23,859 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 14:33:23,859 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:33:23,859 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15307.39 MB 2025-02-14 14:33:23,859 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16024.83 MB 2025-02-14 14:33:23,859 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 717.44 MB 2025-02-14 14:33:23,859 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17379.10 MB 2025-02-14 14:33:23,859 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18587.06 MB 2025-02-14 14:33:23,859 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1207.96 MB 2025-02-14 14:33:23,859 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17798.96 MB 2025-02-14 14:33:23,860 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 14:33:23,860 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 14:33:23,860 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 14:33:23,860 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:33:23,860 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14702.89 MB 2025-02-14 14:33:23,860 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16024.83 MB 2025-02-14 14:33:23,860 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1321.94 MB 2025-02-14 14:33:23,860 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17379.10 MB 2025-02-14 14:33:23,860 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18587.06 MB 2025-02-14 14:33:23,860 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1207.96 MB 2025-02-14 14:33:23,860 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17798.96 MB 2025-02-14 14:33:23,918 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 14:33:23,918 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 14:33:23,918 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-14 14:33:23,918 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:33:23,918 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16515.56 MB 2025-02-14 14:33:23,918 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16761.00 MB 2025-02-14 14:33:23,918 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 245.44 MB 2025-02-14 14:33:23,918 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18587.06 MB 2025-02-14 14:33:23,918 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18719.18 MB 2025-02-14 14:33:23,918 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 132.12 MB 2025-02-14 14:33:23,918 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17000.09 MB 2025-02-14 14:33:23,926 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 14:33:23,926 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 14:33:23,926 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 14:33:23,926 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:33:23,926 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16893.14 MB 2025-02-14 14:33:23,926 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17089.40 MB 2025-02-14 14:33:23,926 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 196.27 MB 2025-02-14 14:33:23,926 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18719.18 MB 2025-02-14 14:33:23,926 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18719.18 MB 2025-02-14 14:33:23,926 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:33:23,926 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17089.40 MB 2025-02-14 14:33:23,928 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 14:33:23,928 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 14:33:23,928 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.75 seconds 2025-02-14 14:33:23,928 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:33:23,928 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13414.67 MB 2025-02-14 14:33:23,928 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17257.62 MB 2025-02-14 14:33:23,928 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3842.95 MB 2025-02-14 14:33:23,928 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56853.79 MB 2025-02-14 14:33:23,928 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18719.18 MB 2025-02-14 14:33:23,928 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -38134.61 MB 2025-02-14 14:33:23,928 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17257.62 MB 2025-02-14 14:33:24,160 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 14:33:24,160 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 14:33:24,160 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 14:33:24,160 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:33:24,160 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17257.62 MB 2025-02-14 14:33:24,160 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16626.46 MB 2025-02-14 14:33:24,160 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -631.16 MB 2025-02-14 14:33:24,160 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18719.18 MB 2025-02-14 14:33:24,160 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19172.16 MB 2025-02-14 14:33:24,160 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 452.98 MB 2025-02-14 14:33:24,160 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18434.32 MB 2025-02-14 14:33:24,176 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 6826, cut from 6828 2025-02-14 14:33:24,176 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 14:33:24,181 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 14:33:24,181 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 14:33:24,181 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:33:24,181 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:33:24,181 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16626.46 MB 2025-02-14 14:33:24,181 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23686.14 MB 2025-02-14 14:33:24,181 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7059.68 MB 2025-02-14 14:33:24,181 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19172.16 MB 2025-02-14 14:33:24,181 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27948.74 MB 2025-02-14 14:33:24,181 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8776.58 MB 2025-02-14 14:33:24,181 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23686.14 MB 2025-02-14 14:33:24,318 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 6618] 2025-02-14 14:33:24,320 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:33:24,320 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 14:33:24,320 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:33:24,321 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 14:33:24,325 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 14:33:24,326 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:33:24,326 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 14:33:24,327 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 14:33:36,567 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:33:36,567 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 14:33:36,572 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 14:33:36,576 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:33:36,576 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1199, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 14:33:36,577 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:33:36,577 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1199, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 14:33:55,066 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 14:33:55,066 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 14:33:55,066 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.48 seconds 2025-02-14 14:33:55,066 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:33:55,066 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21323.53 MB 2025-02-14 14:33:55,066 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25566.72 MB 2025-02-14 14:33:55,066 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4243.19 MB 2025-02-14 14:33:55,066 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34970.01 MB 2025-02-14 14:33:55,066 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35196.50 MB 2025-02-14 14:33:55,066 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 226.49 MB 2025-02-14 14:33:55,066 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34419.59 MB 2025-02-14 14:33:55,133 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 14:33:55,133 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 14:33:55,133 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 14:33:55,133 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:33:55,133 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25566.72 MB 2025-02-14 14:33:55,133 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22011.08 MB 2025-02-14 14:33:55,133 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3555.65 MB 2025-02-14 14:33:55,133 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35196.50 MB 2025-02-14 14:33:55,133 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42867.88 MB 2025-02-14 14:33:55,133 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7671.38 MB 2025-02-14 14:33:55,133 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37514.67 MB 2025-02-14 14:33:57,059 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 14:33:57,060 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 14:33:57,060 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 14:33:57,060 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:33:57,060 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22011.08 MB 2025-02-14 14:33:57,060 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22541.92 MB 2025-02-14 14:33:57,060 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 14:33:57,060 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42867.88 MB 2025-02-14 14:33:57,060 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27441.23 MB 2025-02-14 14:33:57,060 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15426.65 MB 2025-02-14 14:33:57,060 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26521.25 MB 2025-02-14 14:33:57,075 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 14:33:57,075 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 14:33:57,075 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 14:33:57,075 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:33:57,075 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22541.92 MB 2025-02-14 14:33:57,075 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24431.45 MB 2025-02-14 14:33:57,075 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 14:33:57,075 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27441.23 MB 2025-02-14 14:33:57,075 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28384.95 MB 2025-02-14 14:33:57,075 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 14:33:57,075 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25848.88 MB 2025-02-14 14:33:57,285 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 14:33:57,285 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 14:33:57,285 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 14:33:57,285 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:33:57,285 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24431.45 MB 2025-02-14 14:33:57,285 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26673.31 MB 2025-02-14 14:33:57,285 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 14:33:57,285 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28384.95 MB 2025-02-14 14:33:57,286 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34519.12 MB 2025-02-14 14:33:57,286 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 14:33:57,286 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32217.59 MB 2025-02-14 14:33:57,286 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 14:33:57,286 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 14:33:57,286 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 14:33:57,286 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:33:57,286 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22541.92 MB 2025-02-14 14:33:57,286 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26673.31 MB 2025-02-14 14:33:57,286 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 14:33:57,286 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27441.23 MB 2025-02-14 14:33:57,286 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34519.12 MB 2025-02-14 14:33:57,286 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7077.89 MB 2025-02-14 14:33:57,286 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32217.59 MB 2025-02-14 14:33:57,455 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 14:33:57,455 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 14:33:57,455 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 14:33:57,455 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:33:57,455 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28206.85 MB 2025-02-14 14:33:57,455 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28973.85 MB 2025-02-14 14:33:57,455 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 14:33:57,455 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34519.12 MB 2025-02-14 14:33:57,455 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34934.36 MB 2025-02-14 14:33:57,455 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 14:33:57,455 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29681.64 MB 2025-02-14 14:33:57,474 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 14:33:57,474 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 14:33:57,474 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:33:57,474 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:33:57,474 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29386.74 MB 2025-02-14 14:33:57,474 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29615.63 MB 2025-02-14 14:33:57,474 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.89 MB 2025-02-14 14:33:57,474 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34934.36 MB 2025-02-14 14:33:57,474 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34934.36 MB 2025-02-14 14:33:57,474 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:33:57,474 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29817.93 MB 2025-02-14 14:33:57,475 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 14:33:57,476 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 14:33:57,476 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.90 seconds 2025-02-14 14:33:57,476 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:33:57,476 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17146.12 MB 2025-02-14 14:33:57,476 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29816.43 MB 2025-02-14 14:33:57,476 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12670.31 MB 2025-02-14 14:33:57,476 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34970.01 MB 2025-02-14 14:33:57,476 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34934.36 MB 2025-02-14 14:33:57,476 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35.65 MB 2025-02-14 14:33:57,476 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29817.93 MB 2025-02-14 14:33:57,745 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 14:33:57,745 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 14:33:57,745 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 14:33:57,745 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:33:57,745 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29816.43 MB 2025-02-14 14:33:57,745 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22146.32 MB 2025-02-14 14:33:57,745 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7670.11 MB 2025-02-14 14:33:57,745 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34934.36 MB 2025-02-14 14:33:57,745 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34934.36 MB 2025-02-14 14:33:57,745 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:33:57,745 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32324.72 MB 2025-02-14 14:33:57,764 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8151, cut from 8153 2025-02-14 14:33:57,764 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 14:33:57,770 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 14:33:57,770 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 14:33:57,770 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:33:57,770 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:33:57,770 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22146.32 MB 2025-02-14 14:33:57,770 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30573.65 MB 2025-02-14 14:33:57,770 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8427.34 MB 2025-02-14 14:33:57,771 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34934.36 MB 2025-02-14 14:33:57,771 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43314.58 MB 2025-02-14 14:33:57,771 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8380.22 MB 2025-02-14 14:33:57,771 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30573.65 MB 2025-02-14 14:33:57,933 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7943] 2025-02-14 14:33:57,934 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:33:57,934 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 14:33:57,935 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:33:57,935 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 14:33:57,940 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 14:33:57,941 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:33:57,941 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 14:33:57,941 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 14:35:30,716 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:35:30,716 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 14:35:30,724 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 14:35:30,730 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:35:30,731 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 211, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 14:35:30,732 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:35:30,733 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 211, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 14:35:34,090 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 14:35:34,091 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 14:35:34,091 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.35 seconds 2025-02-14 14:35:34,091 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:35:34,091 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14438.99 MB 2025-02-14 14:35:34,091 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15185.71 MB 2025-02-14 14:35:34,091 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 746.72 MB 2025-02-14 14:35:34,091 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51694.80 MB 2025-02-14 14:35:34,091 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17381.20 MB 2025-02-14 14:35:34,091 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34313.60 MB 2025-02-14 14:35:34,091 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24137.66 MB 2025-02-14 14:35:34,113 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 14:35:34,113 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 14:35:34,113 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:35:34,113 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:35:34,113 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15185.71 MB 2025-02-14 14:35:34,113 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15471.22 MB 2025-02-14 14:35:34,113 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 285.51 MB 2025-02-14 14:35:34,113 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17381.20 MB 2025-02-14 14:35:34,113 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19142.80 MB 2025-02-14 14:35:34,113 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1761.61 MB 2025-02-14 14:35:34,113 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18008.64 MB 2025-02-14 14:35:35,086 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 14:35:35,086 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 14:35:35,086 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.97 seconds 2025-02-14 14:35:35,086 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:35:35,086 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15471.22 MB 2025-02-14 14:35:35,086 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15736.64 MB 2025-02-14 14:35:35,086 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 265.42 MB 2025-02-14 14:35:35,086 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19142.80 MB 2025-02-14 14:35:35,086 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18090.03 MB 2025-02-14 14:35:35,086 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1052.77 MB 2025-02-14 14:35:35,086 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19727.63 MB 2025-02-14 14:35:35,095 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 14:35:35,095 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 14:35:35,095 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 14:35:35,095 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:35:35,095 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15736.64 MB 2025-02-14 14:35:35,095 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16681.18 MB 2025-02-14 14:35:35,095 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 944.54 MB 2025-02-14 14:35:35,095 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18090.03 MB 2025-02-14 14:35:35,095 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18561.89 MB 2025-02-14 14:35:35,095 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 471.86 MB 2025-02-14 14:35:35,095 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17389.90 MB 2025-02-14 14:35:35,201 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 14:35:35,201 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 14:35:35,201 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 14:35:35,201 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:35:35,201 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16681.18 MB 2025-02-14 14:35:35,201 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17802.14 MB 2025-02-14 14:35:35,201 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1120.96 MB 2025-02-14 14:35:35,201 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18561.89 MB 2025-02-14 14:35:35,201 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21867.00 MB 2025-02-14 14:35:35,202 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3305.11 MB 2025-02-14 14:35:35,202 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20576.34 MB 2025-02-14 14:35:35,202 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 14:35:35,202 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 14:35:35,202 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 14:35:35,202 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:35:35,202 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15736.64 MB 2025-02-14 14:35:35,202 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17802.14 MB 2025-02-14 14:35:35,202 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2065.50 MB 2025-02-14 14:35:35,202 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18090.03 MB 2025-02-14 14:35:35,202 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21867.00 MB 2025-02-14 14:35:35,202 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3776.97 MB 2025-02-14 14:35:35,202 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20576.34 MB 2025-02-14 14:35:35,287 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 14:35:35,287 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 14:35:35,287 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 14:35:35,287 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:35:35,287 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18569.96 MB 2025-02-14 14:35:35,287 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18954.51 MB 2025-02-14 14:35:35,287 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 384.55 MB 2025-02-14 14:35:35,287 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21867.00 MB 2025-02-14 14:35:35,287 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22072.52 MB 2025-02-14 14:35:35,287 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 205.52 MB 2025-02-14 14:35:35,287 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19311.89 MB 2025-02-14 14:35:35,298 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 14:35:35,298 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 14:35:35,298 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 14:35:35,298 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:35:35,298 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19160.96 MB 2025-02-14 14:35:35,298 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19389.77 MB 2025-02-14 14:35:35,298 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.81 MB 2025-02-14 14:35:35,298 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22072.52 MB 2025-02-14 14:35:35,298 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22072.52 MB 2025-02-14 14:35:35,298 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:35:35,298 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19444.63 MB 2025-02-14 14:35:35,299 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 14:35:35,299 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 14:35:35,299 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.56 seconds 2025-02-14 14:35:35,299 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:35:35,299 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13703.85 MB 2025-02-14 14:35:35,299 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19590.52 MB 2025-02-14 14:35:35,299 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5886.67 MB 2025-02-14 14:35:35,299 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51694.80 MB 2025-02-14 14:35:35,299 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22072.52 MB 2025-02-14 14:35:35,299 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29622.27 MB 2025-02-14 14:35:35,299 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19590.52 MB 2025-02-14 14:35:35,564 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 14:35:35,564 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 14:35:35,564 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 14:35:35,564 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:35:35,564 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14752.28 MB 2025-02-14 14:35:35,564 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17761.52 MB 2025-02-14 14:35:35,564 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3009.24 MB 2025-02-14 14:35:35,564 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22072.52 MB 2025-02-14 14:35:35,564 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22072.52 MB 2025-02-14 14:35:35,564 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:35:35,564 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18062.41 MB 2025-02-14 14:35:35,582 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8149, cut from 8151 2025-02-14 14:35:35,582 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2 ('] 2025-02-14 14:35:35,588 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 14:35:35,588 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 14:35:35,588 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:35:35,589 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:35:35,589 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17761.52 MB 2025-02-14 14:35:35,589 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26187.70 MB 2025-02-14 14:35:35,589 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8426.18 MB 2025-02-14 14:35:35,589 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22072.52 MB 2025-02-14 14:35:35,589 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32543.60 MB 2025-02-14 14:35:35,589 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10471.08 MB 2025-02-14 14:35:35,589 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26187.70 MB 2025-02-14 14:35:35,744 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7941] 2025-02-14 14:35:35,745 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:35:35,745 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 14:35:35,746 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:35:35,746 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 14:35:35,751 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 14:35:35,752 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:35:35,752 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 14:35:35,752 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2 ('] 2025-02-14 14:35:45,707 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:35:45,707 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 14:35:45,712 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 14:35:45,715 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:35:45,715 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2253, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 14:35:45,716 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:35:45,716 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2253, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 14:36:20,575 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 14:36:20,575 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 14:36:20,575 - resource_logging.py:150 - __exit__ - DEBUG - Time: 34.85 seconds 2025-02-14 14:36:20,575 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:36:20,575 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28667.97 MB 2025-02-14 14:36:20,575 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36641.34 MB 2025-02-14 14:36:20,575 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7973.37 MB 2025-02-14 14:36:20,575 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40919.63 MB 2025-02-14 14:36:20,575 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41632.66 MB 2025-02-14 14:36:20,575 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 713.03 MB 2025-02-14 14:36:20,576 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45614.40 MB 2025-02-14 14:36:20,717 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 14:36:20,717 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 14:36:20,717 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-14 14:36:20,717 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:36:20,717 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36641.34 MB 2025-02-14 14:36:20,717 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27490.49 MB 2025-02-14 14:36:20,717 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9150.85 MB 2025-02-14 14:36:20,717 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41632.66 MB 2025-02-14 14:36:20,717 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 69348.62 MB 2025-02-14 14:36:20,717 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 27715.96 MB 2025-02-14 14:36:20,717 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 58818.52 MB 2025-02-14 14:36:22,679 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 14:36:22,679 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 14:36:22,679 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.96 seconds 2025-02-14 14:36:22,679 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:36:22,679 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27490.49 MB 2025-02-14 14:36:22,679 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28021.34 MB 2025-02-14 14:36:22,679 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 14:36:22,679 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 69348.62 MB 2025-02-14 14:36:22,679 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30886.85 MB 2025-02-14 14:36:22,679 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -38461.77 MB 2025-02-14 14:36:22,679 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32001.71 MB 2025-02-14 14:36:22,693 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 14:36:22,693 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 14:36:22,693 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 14:36:22,693 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:36:22,693 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28021.34 MB 2025-02-14 14:36:22,693 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29910.87 MB 2025-02-14 14:36:22,693 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 14:36:22,693 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30886.85 MB 2025-02-14 14:36:22,693 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34189.87 MB 2025-02-14 14:36:22,693 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 14:36:22,693 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31328.30 MB 2025-02-14 14:36:22,902 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 14:36:22,902 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 14:36:22,902 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 14:36:22,902 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:36:22,902 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29910.87 MB 2025-02-14 14:36:22,902 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32152.73 MB 2025-02-14 14:36:22,902 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 14:36:22,902 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34189.87 MB 2025-02-14 14:36:22,902 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40324.04 MB 2025-02-14 14:36:22,902 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 14:36:22,902 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37697.01 MB 2025-02-14 14:36:22,903 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 14:36:22,903 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 14:36:22,903 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 14:36:22,903 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:36:22,903 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28021.34 MB 2025-02-14 14:36:22,903 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32152.73 MB 2025-02-14 14:36:22,903 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 14:36:22,903 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30886.85 MB 2025-02-14 14:36:22,903 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40324.04 MB 2025-02-14 14:36:22,903 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9437.18 MB 2025-02-14 14:36:22,903 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37697.01 MB 2025-02-14 14:36:23,072 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 14:36:23,072 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 14:36:23,072 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 14:36:23,072 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:36:23,072 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33686.27 MB 2025-02-14 14:36:23,072 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34453.27 MB 2025-02-14 14:36:23,072 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 14:36:23,072 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40324.04 MB 2025-02-14 14:36:23,072 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40741.37 MB 2025-02-14 14:36:23,072 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 14:36:23,072 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35161.06 MB 2025-02-14 14:36:23,091 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 14:36:23,091 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 14:36:23,091 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:36:23,091 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:36:23,091 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34866.16 MB 2025-02-14 14:36:23,091 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35094.72 MB 2025-02-14 14:36:23,091 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.56 MB 2025-02-14 14:36:23,091 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40741.37 MB 2025-02-14 14:36:23,091 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40741.37 MB 2025-02-14 14:36:23,091 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:36:23,091 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35312.93 MB 2025-02-14 14:36:23,092 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 14:36:23,092 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 14:36:23,092 - resource_logging.py:150 - __exit__ - DEBUG - Time: 37.37 seconds 2025-02-14 14:36:23,092 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:36:23,092 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20818.34 MB 2025-02-14 14:36:23,092 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35295.40 MB 2025-02-14 14:36:23,092 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14477.06 MB 2025-02-14 14:36:23,092 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40919.63 MB 2025-02-14 14:36:23,092 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40741.37 MB 2025-02-14 14:36:23,092 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -178.26 MB 2025-02-14 14:36:23,092 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35312.93 MB 2025-02-14 14:36:23,363 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 14:36:23,363 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 14:36:23,363 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 14:36:23,363 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:36:23,363 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35295.40 MB 2025-02-14 14:36:23,363 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25817.48 MB 2025-02-14 14:36:23,363 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9477.91 MB 2025-02-14 14:36:23,363 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40741.37 MB 2025-02-14 14:36:23,363 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40741.37 MB 2025-02-14 14:36:23,363 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:36:23,363 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37802.15 MB 2025-02-14 14:36:23,381 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8146, cut from 8148 2025-02-14 14:36:23,381 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 14:36:23,387 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 14:36:23,387 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 14:36:23,387 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:36:23,387 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:36:23,387 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25817.48 MB 2025-02-14 14:36:23,387 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34239.81 MB 2025-02-14 14:36:23,387 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8422.33 MB 2025-02-14 14:36:23,387 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40741.37 MB 2025-02-14 14:36:23,387 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49115.30 MB 2025-02-14 14:36:23,388 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8373.93 MB 2025-02-14 14:36:23,388 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34239.81 MB 2025-02-14 14:36:23,549 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7938] 2025-02-14 14:36:23,551 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:36:23,551 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 14:36:23,552 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:36:23,552 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 14:36:23,556 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 14:36:23,557 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:36:23,558 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 14:36:23,558 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 14:37:14,353 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:37:14,353 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 14:37:14,358 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 14:37:14,361 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:37:14,361 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 209, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 14:37:14,362 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:37:14,362 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 209, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 14:37:17,649 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 14:37:17,649 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 14:37:17,649 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.28 seconds 2025-02-14 14:37:17,649 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:37:17,649 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14425.05 MB 2025-02-14 14:37:17,649 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15164.69 MB 2025-02-14 14:37:17,649 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 739.64 MB 2025-02-14 14:37:17,649 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61675.14 MB 2025-02-14 14:37:17,649 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17853.05 MB 2025-02-14 14:37:17,649 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -43822.09 MB 2025-02-14 14:37:17,649 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24123.72 MB 2025-02-14 14:37:17,690 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 14:37:17,690 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 14:37:17,690 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-14 14:37:17,690 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:37:17,690 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15164.69 MB 2025-02-14 14:37:17,690 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15523.63 MB 2025-02-14 14:37:17,690 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 358.94 MB 2025-02-14 14:37:17,690 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17853.05 MB 2025-02-14 14:37:17,690 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19333.64 MB 2025-02-14 14:37:17,690 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1480.59 MB 2025-02-14 14:37:17,691 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18105.36 MB 2025-02-14 14:37:18,703 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 14:37:18,703 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 14:37:18,703 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.01 seconds 2025-02-14 14:37:18,703 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:37:18,703 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15523.63 MB 2025-02-14 14:37:18,703 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15801.00 MB 2025-02-14 14:37:18,703 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 277.36 MB 2025-02-14 14:37:18,703 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19333.64 MB 2025-02-14 14:37:18,703 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18524.14 MB 2025-02-14 14:37:18,703 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -809.50 MB 2025-02-14 14:37:18,703 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19780.04 MB 2025-02-14 14:37:18,713 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 14:37:18,713 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 14:37:18,713 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 14:37:18,713 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:37:18,713 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15801.00 MB 2025-02-14 14:37:18,713 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16788.04 MB 2025-02-14 14:37:18,713 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 987.04 MB 2025-02-14 14:37:18,713 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18524.14 MB 2025-02-14 14:37:18,713 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19019.07 MB 2025-02-14 14:37:18,713 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 494.93 MB 2025-02-14 14:37:18,713 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17528.65 MB 2025-02-14 14:37:18,824 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 14:37:18,824 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 14:37:18,824 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 14:37:18,824 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:37:18,824 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16788.04 MB 2025-02-14 14:37:18,824 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17959.67 MB 2025-02-14 14:37:18,824 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1171.63 MB 2025-02-14 14:37:18,824 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19019.07 MB 2025-02-14 14:37:18,824 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21988.64 MB 2025-02-14 14:37:18,824 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2969.57 MB 2025-02-14 14:37:18,824 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20856.53 MB 2025-02-14 14:37:18,825 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 14:37:18,825 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 14:37:18,825 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 14:37:18,825 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:37:18,825 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15801.00 MB 2025-02-14 14:37:18,825 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17959.67 MB 2025-02-14 14:37:18,825 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2158.67 MB 2025-02-14 14:37:18,825 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18524.14 MB 2025-02-14 14:37:18,825 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21988.64 MB 2025-02-14 14:37:18,825 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3464.50 MB 2025-02-14 14:37:18,825 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20856.53 MB 2025-02-14 14:37:18,910 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 14:37:18,910 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 14:37:18,910 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 14:37:18,910 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:37:18,910 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18760.95 MB 2025-02-14 14:37:18,910 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19161.71 MB 2025-02-14 14:37:18,910 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 400.76 MB 2025-02-14 14:37:18,910 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21988.64 MB 2025-02-14 14:37:18,910 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22204.65 MB 2025-02-14 14:37:18,910 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 216.01 MB 2025-02-14 14:37:18,910 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19533.59 MB 2025-02-14 14:37:18,921 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 14:37:18,921 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 14:37:18,921 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 14:37:18,922 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:37:18,922 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19377.45 MB 2025-02-14 14:37:18,922 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19606.27 MB 2025-02-14 14:37:18,922 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.82 MB 2025-02-14 14:37:18,922 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22204.65 MB 2025-02-14 14:37:18,922 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22204.65 MB 2025-02-14 14:37:18,922 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:37:18,922 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19671.49 MB 2025-02-14 14:37:18,923 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 14:37:18,923 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 14:37:18,923 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.56 seconds 2025-02-14 14:37:18,923 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:37:18,923 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13696.88 MB 2025-02-14 14:37:18,923 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19807.34 MB 2025-02-14 14:37:18,923 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6110.46 MB 2025-02-14 14:37:18,923 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61675.14 MB 2025-02-14 14:37:18,923 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22204.65 MB 2025-02-14 14:37:18,923 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -39470.50 MB 2025-02-14 14:37:18,923 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19807.34 MB 2025-02-14 14:37:19,189 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 14:37:19,189 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 14:37:19,189 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 14:37:19,189 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:37:19,189 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14785.59 MB 2025-02-14 14:37:19,189 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17800.41 MB 2025-02-14 14:37:19,189 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.82 MB 2025-02-14 14:37:19,189 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22204.65 MB 2025-02-14 14:37:19,189 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22204.65 MB 2025-02-14 14:37:19,189 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:37:19,189 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18101.77 MB 2025-02-14 14:37:19,207 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 14:37:19,207 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 14:37:19,213 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 14:37:19,213 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 14:37:19,213 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:37:19,213 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:37:19,213 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17800.41 MB 2025-02-14 14:37:19,213 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26239.43 MB 2025-02-14 14:37:19,213 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 14:37:19,213 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22204.65 MB 2025-02-14 14:37:19,213 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32694.60 MB 2025-02-14 14:37:19,213 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 14:37:19,213 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26239.43 MB 2025-02-14 14:37:19,365 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 14:37:19,366 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:37:19,366 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 14:37:19,367 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:37:19,367 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 14:37:19,371 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 14:37:19,373 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:37:19,373 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 14:37:19,373 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 14:38:37,759 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:38:37,760 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 14:38:37,765 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 14:38:37,769 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:38:37,769 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1421, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 14:38:37,770 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:38:37,770 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1421, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 14:38:59,603 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 14:38:59,603 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 14:38:59,603 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.82 seconds 2025-02-14 14:38:59,603 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:38:59,603 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22870.46 MB 2025-02-14 14:38:59,603 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27899.43 MB 2025-02-14 14:38:59,603 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5028.97 MB 2025-02-14 14:38:59,603 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45279.61 MB 2025-02-14 14:38:59,603 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38715.52 MB 2025-02-14 14:38:59,603 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6564.09 MB 2025-02-14 14:38:59,603 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36871.68 MB 2025-02-14 14:38:59,686 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 14:38:59,686 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 14:38:59,686 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 14:38:59,686 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:38:59,686 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27899.43 MB 2025-02-14 14:38:59,686 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23165.19 MB 2025-02-14 14:38:59,686 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4734.25 MB 2025-02-14 14:38:59,686 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38715.52 MB 2025-02-14 14:38:59,686 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48326.77 MB 2025-02-14 14:38:59,686 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9611.25 MB 2025-02-14 14:38:59,686 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42395.46 MB 2025-02-14 14:39:01,595 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 14:39:01,595 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 14:39:01,595 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 14:39:01,596 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:39:01,596 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23165.19 MB 2025-02-14 14:39:01,596 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23696.03 MB 2025-02-14 14:39:01,596 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 14:39:01,596 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48326.77 MB 2025-02-14 14:39:01,596 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33686.55 MB 2025-02-14 14:39:01,596 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14640.22 MB 2025-02-14 14:39:01,596 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27675.36 MB 2025-02-14 14:39:01,609 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 14:39:01,609 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 14:39:01,609 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 14:39:01,609 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:39:01,609 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23696.03 MB 2025-02-14 14:39:01,609 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25585.56 MB 2025-02-14 14:39:01,609 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 14:39:01,609 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33686.55 MB 2025-02-14 14:39:01,609 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33686.55 MB 2025-02-14 14:39:01,609 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:39:01,609 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27002.99 MB 2025-02-14 14:39:01,822 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 14:39:01,822 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 14:39:01,822 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 14:39:01,822 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:39:01,822 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25585.56 MB 2025-02-14 14:39:01,823 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27827.42 MB 2025-02-14 14:39:01,823 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 14:39:01,823 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33686.55 MB 2025-02-14 14:39:01,823 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37461.43 MB 2025-02-14 14:39:01,823 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 14:39:01,823 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33371.70 MB 2025-02-14 14:39:01,823 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 14:39:01,823 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 14:39:01,823 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 14:39:01,823 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:39:01,823 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23696.03 MB 2025-02-14 14:39:01,823 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27827.42 MB 2025-02-14 14:39:01,823 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 14:39:01,823 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33686.55 MB 2025-02-14 14:39:01,823 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37461.43 MB 2025-02-14 14:39:01,823 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 14:39:01,823 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33371.70 MB 2025-02-14 14:39:02,001 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 14:39:02,001 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 14:39:02,001 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 14:39:02,001 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:39:02,001 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29360.96 MB 2025-02-14 14:39:02,001 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30127.96 MB 2025-02-14 14:39:02,001 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 14:39:02,002 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37461.43 MB 2025-02-14 14:39:02,002 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37878.76 MB 2025-02-14 14:39:02,002 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 14:39:02,002 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30835.75 MB 2025-02-14 14:39:02,021 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 14:39:02,021 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 14:39:02,021 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:39:02,021 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:39:02,021 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30540.85 MB 2025-02-14 14:39:02,021 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30769.52 MB 2025-02-14 14:39:02,021 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.67 MB 2025-02-14 14:39:02,021 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37878.76 MB 2025-02-14 14:39:02,021 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37878.76 MB 2025-02-14 14:39:02,021 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:39:02,021 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30981.42 MB 2025-02-14 14:39:02,022 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 14:39:02,022 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 14:39:02,022 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.25 seconds 2025-02-14 14:39:02,022 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:39:02,022 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17919.58 MB 2025-02-14 14:39:02,022 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30970.10 MB 2025-02-14 14:39:02,022 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13050.51 MB 2025-02-14 14:39:02,022 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45279.61 MB 2025-02-14 14:39:02,022 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37878.76 MB 2025-02-14 14:39:02,022 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7400.85 MB 2025-02-14 14:39:02,022 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30981.42 MB 2025-02-14 14:39:02,292 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 14:39:02,292 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 14:39:02,292 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 14:39:02,292 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:39:02,292 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30970.10 MB 2025-02-14 14:39:02,292 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22916.35 MB 2025-02-14 14:39:02,292 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8053.74 MB 2025-02-14 14:39:02,292 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37878.76 MB 2025-02-14 14:39:02,292 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37878.76 MB 2025-02-14 14:39:02,292 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:39:02,292 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33475.62 MB 2025-02-14 14:39:02,310 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8142, cut from 8144 2025-02-14 14:39:02,310 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-14 14:39:02,316 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 14:39:02,316 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 14:39:02,316 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:39:02,316 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:39:02,316 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22916.35 MB 2025-02-14 14:39:02,316 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31334.51 MB 2025-02-14 14:39:02,316 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8418.15 MB 2025-02-14 14:39:02,316 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37878.76 MB 2025-02-14 14:39:02,316 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42062.58 MB 2025-02-14 14:39:02,316 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4183.82 MB 2025-02-14 14:39:02,316 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31334.51 MB 2025-02-14 14:39:02,479 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7934] 2025-02-14 14:39:02,480 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:39:02,480 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 14:39:02,481 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:39:02,481 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 14:39:02,486 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 14:39:02,487 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:39:02,487 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 14:39:02,487 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-14 14:40:27,537 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:40:27,538 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 14:40:27,542 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 14:40:27,547 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:40:27,547 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1808, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 14:40:27,548 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:40:27,548 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1808, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 14:40:55,452 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 14:40:55,453 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 14:40:55,453 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.90 seconds 2025-02-14 14:40:55,453 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:40:55,453 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25567.14 MB 2025-02-14 14:40:55,453 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31965.55 MB 2025-02-14 14:40:55,453 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6398.41 MB 2025-02-14 14:40:55,453 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54616.13 MB 2025-02-14 14:40:55,453 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40057.70 MB 2025-02-14 14:40:55,453 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14558.43 MB 2025-02-14 14:40:55,453 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40928.12 MB 2025-02-14 14:40:55,562 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 14:40:55,562 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 14:40:55,562 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 14:40:55,562 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:40:55,562 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31965.55 MB 2025-02-14 14:40:55,562 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25177.08 MB 2025-02-14 14:40:55,562 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6788.47 MB 2025-02-14 14:40:55,562 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40057.70 MB 2025-02-14 14:40:55,562 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58716.06 MB 2025-02-14 14:40:55,562 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 18658.36 MB 2025-02-14 14:40:55,562 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49789.24 MB 2025-02-14 14:40:57,493 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 14:40:57,493 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 14:40:57,493 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 14:40:57,493 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:40:57,493 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25177.08 MB 2025-02-14 14:40:57,493 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25707.92 MB 2025-02-14 14:40:57,493 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 14:40:57,493 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58716.06 MB 2025-02-14 14:40:57,493 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35074.87 MB 2025-02-14 14:40:57,493 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23641.19 MB 2025-02-14 14:40:57,493 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29687.25 MB 2025-02-14 14:40:57,507 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 14:40:57,507 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 14:40:57,507 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 14:40:57,507 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:40:57,507 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25707.92 MB 2025-02-14 14:40:57,507 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27597.45 MB 2025-02-14 14:40:57,507 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 14:40:57,507 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35074.87 MB 2025-02-14 14:40:57,507 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35074.87 MB 2025-02-14 14:40:57,507 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:40:57,507 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29014.88 MB 2025-02-14 14:40:57,725 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 14:40:57,725 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 14:40:57,725 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 14:40:57,725 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:40:57,725 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27597.45 MB 2025-02-14 14:40:57,725 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29839.31 MB 2025-02-14 14:40:57,725 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 14:40:57,725 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35074.87 MB 2025-02-14 14:40:57,725 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38377.88 MB 2025-02-14 14:40:57,725 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 14:40:57,725 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35383.59 MB 2025-02-14 14:40:57,726 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 14:40:57,726 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 14:40:57,726 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 14:40:57,726 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:40:57,726 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25707.92 MB 2025-02-14 14:40:57,726 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29839.31 MB 2025-02-14 14:40:57,726 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 14:40:57,726 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35074.87 MB 2025-02-14 14:40:57,726 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38377.88 MB 2025-02-14 14:40:57,726 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 14:40:57,726 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35383.59 MB 2025-02-14 14:40:57,900 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 14:40:57,900 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 14:40:57,900 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 14:40:57,900 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:40:57,900 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31372.85 MB 2025-02-14 14:40:57,900 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32139.85 MB 2025-02-14 14:40:57,900 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 14:40:57,900 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38377.88 MB 2025-02-14 14:40:57,900 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38791.02 MB 2025-02-14 14:40:57,900 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 14:40:57,900 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32847.64 MB 2025-02-14 14:40:57,920 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 14:40:57,920 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 14:40:57,920 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:40:57,920 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:40:57,920 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32552.74 MB 2025-02-14 14:40:57,920 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32780.84 MB 2025-02-14 14:40:57,920 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.10 MB 2025-02-14 14:40:57,920 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38791.02 MB 2025-02-14 14:40:57,920 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38791.02 MB 2025-02-14 14:40:57,920 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:40:57,920 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33002.31 MB 2025-02-14 14:40:57,921 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 14:40:57,921 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 14:40:57,921 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.37 seconds 2025-02-14 14:40:57,921 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:40:57,921 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19267.92 MB 2025-02-14 14:40:57,921 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32980.86 MB 2025-02-14 14:40:57,921 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13712.94 MB 2025-02-14 14:40:57,921 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54616.13 MB 2025-02-14 14:40:57,921 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38791.02 MB 2025-02-14 14:40:57,921 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15825.11 MB 2025-02-14 14:40:57,921 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33002.31 MB 2025-02-14 14:40:58,192 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 14:40:58,192 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 14:40:58,192 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 14:40:58,192 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:40:58,192 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32980.86 MB 2025-02-14 14:40:58,192 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24255.93 MB 2025-02-14 14:40:58,192 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8724.93 MB 2025-02-14 14:40:58,192 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38791.02 MB 2025-02-14 14:40:58,192 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38791.02 MB 2025-02-14 14:40:58,192 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:40:58,192 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35479.32 MB 2025-02-14 14:40:58,210 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8119, cut from 8121 2025-02-14 14:40:58,210 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 14:40:58,216 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 14:40:58,216 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 14:40:58,216 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:40:58,216 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:40:58,216 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24255.93 MB 2025-02-14 14:40:58,216 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32651.15 MB 2025-02-14 14:40:58,216 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8395.21 MB 2025-02-14 14:40:58,216 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38791.02 MB 2025-02-14 14:40:58,216 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47137.69 MB 2025-02-14 14:40:58,216 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8346.66 MB 2025-02-14 14:40:58,216 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32651.15 MB 2025-02-14 14:40:58,379 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7911] 2025-02-14 14:40:58,381 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:40:58,381 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 14:40:58,382 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:40:58,382 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 14:40:58,386 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 14:40:58,387 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:40:58,387 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 14:40:58,388 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 14:41:10,401 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:41:10,401 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 14:41:10,406 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 14:41:10,410 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:41:10,410 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2330, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 14:41:10,411 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:41:10,411 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2330, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 14:41:46,936 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 14:41:46,936 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 14:41:46,936 - resource_logging.py:150 - __exit__ - DEBUG - Time: 36.52 seconds 2025-02-14 14:41:46,936 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:41:46,936 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29204.52 MB 2025-02-14 14:41:46,936 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37450.52 MB 2025-02-14 14:41:46,936 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8246.00 MB 2025-02-14 14:41:46,936 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55484.35 MB 2025-02-14 14:41:46,936 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41848.67 MB 2025-02-14 14:41:46,936 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13635.68 MB 2025-02-14 14:41:46,936 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46377.44 MB 2025-02-14 14:41:47,061 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 14:41:47,061 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 14:41:47,061 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 14:41:47,061 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:41:47,061 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37450.52 MB 2025-02-14 14:41:47,061 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27890.79 MB 2025-02-14 14:41:47,061 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9559.73 MB 2025-02-14 14:41:47,061 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41848.67 MB 2025-02-14 14:41:47,061 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 60462.99 MB 2025-02-14 14:41:47,061 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 18614.32 MB 2025-02-14 14:41:47,061 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52424.27 MB 2025-02-14 14:41:49,040 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 14:41:49,040 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 14:41:49,040 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.98 seconds 2025-02-14 14:41:49,040 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:41:49,040 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27890.79 MB 2025-02-14 14:41:49,040 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28421.64 MB 2025-02-14 14:41:49,040 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 14:41:49,040 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60462.99 MB 2025-02-14 14:41:49,040 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30844.91 MB 2025-02-14 14:41:49,040 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29618.08 MB 2025-02-14 14:41:49,040 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32402.01 MB 2025-02-14 14:41:49,055 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 14:41:49,055 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 14:41:49,055 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 14:41:49,055 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:41:49,055 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28421.64 MB 2025-02-14 14:41:49,055 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30311.17 MB 2025-02-14 14:41:49,055 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 14:41:49,055 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30844.91 MB 2025-02-14 14:41:49,055 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34147.93 MB 2025-02-14 14:41:49,055 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 14:41:49,055 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31728.60 MB 2025-02-14 14:41:49,268 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 14:41:49,268 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 14:41:49,268 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 14:41:49,268 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:41:49,268 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30311.17 MB 2025-02-14 14:41:49,268 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32553.03 MB 2025-02-14 14:41:49,268 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 14:41:49,268 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34147.93 MB 2025-02-14 14:41:49,268 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40753.95 MB 2025-02-14 14:41:49,268 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 14:41:49,268 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38097.31 MB 2025-02-14 14:41:49,269 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 14:41:49,269 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 14:41:49,269 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 14:41:49,269 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:41:49,269 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28421.64 MB 2025-02-14 14:41:49,269 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32553.03 MB 2025-02-14 14:41:49,269 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 14:41:49,269 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30844.91 MB 2025-02-14 14:41:49,269 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40753.95 MB 2025-02-14 14:41:49,269 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9909.04 MB 2025-02-14 14:41:49,269 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38097.31 MB 2025-02-14 14:41:49,443 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 14:41:49,443 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 14:41:49,443 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 14:41:49,443 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:41:49,443 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34086.57 MB 2025-02-14 14:41:49,443 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34853.57 MB 2025-02-14 14:41:49,443 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 14:41:49,443 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40753.95 MB 2025-02-14 14:41:49,443 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41169.19 MB 2025-02-14 14:41:49,443 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 14:41:49,443 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35561.36 MB 2025-02-14 14:41:49,465 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 14:41:49,465 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 14:41:49,465 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:41:49,465 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:41:49,465 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35266.46 MB 2025-02-14 14:41:49,465 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35495.63 MB 2025-02-14 14:41:49,465 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.17 MB 2025-02-14 14:41:49,465 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41169.19 MB 2025-02-14 14:41:49,465 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41169.19 MB 2025-02-14 14:41:49,465 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:41:49,465 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35730.23 MB 2025-02-14 14:41:49,467 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 14:41:49,467 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 14:41:49,467 - resource_logging.py:150 - __exit__ - DEBUG - Time: 39.05 seconds 2025-02-14 14:41:49,467 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:41:49,467 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21086.61 MB 2025-02-14 14:41:49,467 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35696.70 MB 2025-02-14 14:41:49,467 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14610.09 MB 2025-02-14 14:41:49,467 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55484.35 MB 2025-02-14 14:41:49,467 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41169.19 MB 2025-02-14 14:41:49,467 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14315.16 MB 2025-02-14 14:41:49,467 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35730.23 MB 2025-02-14 14:41:49,739 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 14:41:49,739 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 14:41:49,739 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 14:41:49,739 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:41:49,739 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35696.70 MB 2025-02-14 14:41:49,739 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26091.00 MB 2025-02-14 14:41:49,739 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9605.70 MB 2025-02-14 14:41:49,739 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41169.19 MB 2025-02-14 14:41:49,739 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41169.19 MB 2025-02-14 14:41:49,739 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:41:49,740 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38208.37 MB 2025-02-14 14:41:49,757 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 14:41:49,758 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 14:41:49,764 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 14:41:49,764 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 14:41:49,764 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:41:49,764 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:41:49,764 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26091.00 MB 2025-02-14 14:41:49,764 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34530.25 MB 2025-02-14 14:41:49,764 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.25 MB 2025-02-14 14:41:49,764 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41169.19 MB 2025-02-14 14:41:49,764 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49559.90 MB 2025-02-14 14:41:49,764 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 14:41:49,764 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34530.25 MB 2025-02-14 14:41:49,927 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 14:41:49,928 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:41:49,928 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 14:41:49,929 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:41:49,929 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 14:41:49,934 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 14:41:49,935 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:41:49,935 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 14:41:49,935 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 14:42:37,430 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:42:37,430 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 14:42:37,435 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 14:42:37,438 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:42:37,439 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 237, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 14:42:37,439 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:42:37,440 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 237, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 14:42:41,122 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 14:42:41,122 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 14:42:41,122 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.68 seconds 2025-02-14 14:42:41,122 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:42:41,122 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14621.06 MB 2025-02-14 14:42:41,122 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15459.79 MB 2025-02-14 14:42:41,122 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 838.73 MB 2025-02-14 14:42:41,122 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62144.91 MB 2025-02-14 14:42:41,122 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18794.68 MB 2025-02-14 14:42:41,122 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -43350.23 MB 2025-02-14 14:42:41,122 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24319.73 MB 2025-02-14 14:42:41,140 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 14:42:41,140 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 14:42:41,140 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:42:41,140 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:42:41,140 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15459.79 MB 2025-02-14 14:42:41,140 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15774.79 MB 2025-02-14 14:42:41,140 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 315.00 MB 2025-02-14 14:42:41,140 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18794.68 MB 2025-02-14 14:42:41,140 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20380.12 MB 2025-02-14 14:42:41,140 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1585.45 MB 2025-02-14 14:42:41,140 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18652.11 MB 2025-02-14 14:42:42,216 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 14:42:42,216 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 14:42:42,217 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.07 seconds 2025-02-14 14:42:42,217 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:42:42,217 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15774.79 MB 2025-02-14 14:42:42,217 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16072.06 MB 2025-02-14 14:42:42,217 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 297.27 MB 2025-02-14 14:42:42,217 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20380.12 MB 2025-02-14 14:42:42,217 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19115.54 MB 2025-02-14 14:42:42,217 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1264.58 MB 2025-02-14 14:42:42,217 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20031.20 MB 2025-02-14 14:42:42,234 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 14:42:42,234 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 14:42:42,234 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 14:42:42,234 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:42:42,234 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16072.06 MB 2025-02-14 14:42:42,234 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17129.94 MB 2025-02-14 14:42:42,234 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1057.88 MB 2025-02-14 14:42:42,234 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19115.54 MB 2025-02-14 14:42:42,234 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19644.02 MB 2025-02-14 14:42:42,234 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 528.48 MB 2025-02-14 14:42:42,234 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17923.70 MB 2025-02-14 14:42:42,360 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 14:42:42,360 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 14:42:42,360 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 14:42:42,360 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:42:42,360 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17129.94 MB 2025-02-14 14:42:42,360 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18385.41 MB 2025-02-14 14:42:42,360 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1255.47 MB 2025-02-14 14:42:42,360 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19644.02 MB 2025-02-14 14:42:42,360 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22814.92 MB 2025-02-14 14:42:42,360 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3170.89 MB 2025-02-14 14:42:42,360 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21490.18 MB 2025-02-14 14:42:42,361 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 14:42:42,361 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 14:42:42,361 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-14 14:42:42,361 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:42:42,361 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16072.06 MB 2025-02-14 14:42:42,361 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18385.41 MB 2025-02-14 14:42:42,361 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2313.35 MB 2025-02-14 14:42:42,361 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19115.54 MB 2025-02-14 14:42:42,361 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22814.92 MB 2025-02-14 14:42:42,361 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3699.38 MB 2025-02-14 14:42:42,361 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21490.18 MB 2025-02-14 14:42:42,460 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 14:42:42,460 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 14:42:42,460 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 14:42:42,460 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:42:42,460 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19244.19 MB 2025-02-14 14:42:42,460 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19673.71 MB 2025-02-14 14:42:42,460 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 429.52 MB 2025-02-14 14:42:42,460 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22814.92 MB 2025-02-14 14:42:42,460 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23047.70 MB 2025-02-14 14:42:42,460 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 232.78 MB 2025-02-14 14:42:42,460 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20071.66 MB 2025-02-14 14:42:42,473 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 14:42:42,473 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 14:42:42,473 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 14:42:42,473 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:42:42,473 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19904.94 MB 2025-02-14 14:42:42,473 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20133.11 MB 2025-02-14 14:42:42,473 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.18 MB 2025-02-14 14:42:42,473 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23047.70 MB 2025-02-14 14:42:42,473 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23047.70 MB 2025-02-14 14:42:42,473 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:42:42,473 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20182.41 MB 2025-02-14 14:42:42,474 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 14:42:42,474 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 14:42:42,474 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.03 seconds 2025-02-14 14:42:42,474 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:42:42,474 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13795.33 MB 2025-02-14 14:42:42,474 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20333.84 MB 2025-02-14 14:42:42,474 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6538.51 MB 2025-02-14 14:42:42,474 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62144.91 MB 2025-02-14 14:42:42,474 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23047.70 MB 2025-02-14 14:42:42,474 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -39097.20 MB 2025-02-14 14:42:42,474 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20333.84 MB 2025-02-14 14:42:42,742 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 14:42:42,742 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 14:42:42,742 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 14:42:42,742 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:42:42,742 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14954.92 MB 2025-02-14 14:42:42,742 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17963.79 MB 2025-02-14 14:42:42,742 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3008.87 MB 2025-02-14 14:42:42,742 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23047.70 MB 2025-02-14 14:42:42,742 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23047.70 MB 2025-02-14 14:42:42,742 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:42:42,742 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18264.64 MB 2025-02-14 14:42:42,760 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8148, cut from 8150 2025-02-14 14:42:42,760 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 14:42:42,766 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 14:42:42,766 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 14:42:42,766 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:42:42,766 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:42:42,766 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17963.79 MB 2025-02-14 14:42:42,766 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26388.74 MB 2025-02-14 14:42:42,766 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8424.95 MB 2025-02-14 14:42:42,766 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23047.70 MB 2025-02-14 14:42:42,766 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33518.78 MB 2025-02-14 14:42:42,766 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10471.08 MB 2025-02-14 14:42:42,766 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26388.74 MB 2025-02-14 14:42:42,929 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7940] 2025-02-14 14:42:42,931 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:42:42,931 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 14:42:42,932 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:42:42,932 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 14:42:42,936 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 14:42:42,937 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:42:42,937 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 14:42:42,938 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 14:43:15,191 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:43:15,191 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 14:43:15,197 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 14:43:15,200 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:43:15,200 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1035, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 14:43:15,201 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:43:15,201 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1035, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 14:43:31,227 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 14:43:31,227 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 14:43:31,227 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.02 seconds 2025-02-14 14:43:31,227 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:43:31,227 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20180.75 MB 2025-02-14 14:43:31,227 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23844.48 MB 2025-02-14 14:43:31,227 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3663.72 MB 2025-02-14 14:43:31,227 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41894.81 MB 2025-02-14 14:43:31,227 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31042.04 MB 2025-02-14 14:43:31,227 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10852.76 MB 2025-02-14 14:43:31,227 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32823.02 MB 2025-02-14 14:43:31,288 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 14:43:31,288 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 14:43:31,288 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 14:43:31,288 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:43:31,288 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23844.48 MB 2025-02-14 14:43:31,288 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21158.49 MB 2025-02-14 14:43:31,288 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2685.98 MB 2025-02-14 14:43:31,288 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31042.04 MB 2025-02-14 14:43:31,288 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39197.87 MB 2025-02-14 14:43:31,288 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8155.82 MB 2025-02-14 14:43:31,288 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34843.27 MB 2025-02-14 14:43:33,209 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 14:43:33,209 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 14:43:33,209 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 14:43:33,209 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:43:33,209 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21158.49 MB 2025-02-14 14:43:33,209 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21689.33 MB 2025-02-14 14:43:33,209 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 14:43:33,209 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39197.87 MB 2025-02-14 14:43:33,209 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28793.90 MB 2025-02-14 14:43:33,209 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10403.97 MB 2025-02-14 14:43:33,209 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25668.67 MB 2025-02-14 14:43:33,222 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 14:43:33,222 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 14:43:33,222 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 14:43:33,222 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:43:33,222 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21689.33 MB 2025-02-14 14:43:33,222 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23578.87 MB 2025-02-14 14:43:33,222 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 14:43:33,222 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28793.90 MB 2025-02-14 14:43:33,222 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28793.90 MB 2025-02-14 14:43:33,222 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:43:33,222 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24996.30 MB 2025-02-14 14:43:33,437 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 14:43:33,437 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 14:43:33,437 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 14:43:33,437 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:43:33,437 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23578.87 MB 2025-02-14 14:43:33,437 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25820.72 MB 2025-02-14 14:43:33,437 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 14:43:33,437 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28793.90 MB 2025-02-14 14:43:33,437 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33984.35 MB 2025-02-14 14:43:33,437 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 14:43:33,437 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31365.01 MB 2025-02-14 14:43:33,438 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 14:43:33,438 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 14:43:33,438 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 14:43:33,438 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:43:33,438 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21689.33 MB 2025-02-14 14:43:33,438 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25820.72 MB 2025-02-14 14:43:33,438 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 14:43:33,438 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28793.90 MB 2025-02-14 14:43:33,438 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33984.35 MB 2025-02-14 14:43:33,438 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 14:43:33,438 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31365.01 MB 2025-02-14 14:43:33,608 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 14:43:33,608 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 14:43:33,608 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 14:43:33,608 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:43:33,608 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27354.27 MB 2025-02-14 14:43:33,608 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28121.27 MB 2025-02-14 14:43:33,608 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 14:43:33,608 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33984.35 MB 2025-02-14 14:43:33,608 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34401.68 MB 2025-02-14 14:43:33,608 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 14:43:33,608 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28829.06 MB 2025-02-14 14:43:33,627 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 14:43:33,627 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 14:43:33,627 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:43:33,627 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:43:33,627 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28534.16 MB 2025-02-14 14:43:33,627 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28763.71 MB 2025-02-14 14:43:33,627 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.55 MB 2025-02-14 14:43:33,627 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34401.68 MB 2025-02-14 14:43:33,627 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34401.68 MB 2025-02-14 14:43:33,627 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:43:33,627 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28976.11 MB 2025-02-14 14:43:33,630 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 14:43:33,630 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 14:43:33,630 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.42 seconds 2025-02-14 14:43:33,630 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:43:33,630 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16574.73 MB 2025-02-14 14:43:33,630 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28964.78 MB 2025-02-14 14:43:33,630 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12390.05 MB 2025-02-14 14:43:33,630 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41894.81 MB 2025-02-14 14:43:33,630 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34401.68 MB 2025-02-14 14:43:33,630 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7493.12 MB 2025-02-14 14:43:33,630 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28976.11 MB 2025-02-14 14:43:33,898 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 14:43:33,898 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 14:43:33,898 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 14:43:33,898 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:43:33,898 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28964.78 MB 2025-02-14 14:43:33,898 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21579.12 MB 2025-02-14 14:43:33,898 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7385.66 MB 2025-02-14 14:43:33,898 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34401.68 MB 2025-02-14 14:43:33,898 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34401.68 MB 2025-02-14 14:43:33,898 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:43:33,898 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31476.45 MB 2025-02-14 14:43:33,916 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 14:43:33,916 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 14:43:33,922 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 14:43:33,923 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 14:43:33,923 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:43:33,923 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:43:33,923 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21579.12 MB 2025-02-14 14:43:33,923 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30018.14 MB 2025-02-14 14:43:33,923 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 14:43:33,923 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34401.68 MB 2025-02-14 14:43:33,923 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42792.39 MB 2025-02-14 14:43:33,923 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 14:43:33,923 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30018.14 MB 2025-02-14 14:43:34,088 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 14:43:34,089 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:43:34,089 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 14:43:34,090 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:43:34,090 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 14:43:34,095 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 14:43:34,096 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:43:34,096 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 14:43:34,096 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 14:45:32,477 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:45:32,478 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 14:45:32,485 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 14:45:32,493 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:45:32,493 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 615, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 14:45:32,495 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:45:32,495 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 615, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 14:45:42,069 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 14:45:42,069 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 14:45:42,069 - resource_logging.py:150 - __exit__ - DEBUG - Time: 9.57 seconds 2025-02-14 14:45:42,069 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:45:42,069 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17254.12 MB 2025-02-14 14:45:42,069 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19430.58 MB 2025-02-14 14:45:42,069 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2176.45 MB 2025-02-14 14:45:42,069 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55377.40 MB 2025-02-14 14:45:42,069 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24607.98 MB 2025-02-14 14:45:42,069 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30769.41 MB 2025-02-14 14:45:42,069 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28310.94 MB 2025-02-14 14:45:42,112 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 14:45:42,112 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 14:45:42,112 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-14 14:45:42,112 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:45:42,112 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19430.58 MB 2025-02-14 14:45:42,112 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18975.04 MB 2025-02-14 14:45:42,112 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -455.53 MB 2025-02-14 14:45:42,112 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24607.98 MB 2025-02-14 14:45:42,112 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31759.27 MB 2025-02-14 14:45:42,112 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7151.29 MB 2025-02-14 14:45:42,112 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28000.27 MB 2025-02-14 14:45:44,058 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 14:45:44,058 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 14:45:44,058 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-14 14:45:44,058 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:45:44,058 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18975.04 MB 2025-02-14 14:45:44,058 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19505.89 MB 2025-02-14 14:45:44,058 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 14:45:44,058 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31759.27 MB 2025-02-14 14:45:44,058 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23928.50 MB 2025-02-14 14:45:44,058 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7830.77 MB 2025-02-14 14:45:44,058 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23486.26 MB 2025-02-14 14:45:44,073 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 14:45:44,073 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 14:45:44,073 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 14:45:44,073 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:45:44,073 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19505.89 MB 2025-02-14 14:45:44,073 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21395.42 MB 2025-02-14 14:45:44,073 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 14:45:44,073 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23928.50 MB 2025-02-14 14:45:44,073 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25815.94 MB 2025-02-14 14:45:44,073 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 14:45:44,073 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22812.85 MB 2025-02-14 14:45:44,281 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 14:45:44,281 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 14:45:44,281 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 14:45:44,281 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:45:44,281 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21395.42 MB 2025-02-14 14:45:44,281 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23637.28 MB 2025-02-14 14:45:44,281 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 14:45:44,281 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25815.94 MB 2025-02-14 14:45:44,281 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31950.11 MB 2025-02-14 14:45:44,281 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 14:45:44,281 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29181.56 MB 2025-02-14 14:45:44,282 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 14:45:44,282 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 14:45:44,282 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 14:45:44,282 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:45:44,282 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19505.89 MB 2025-02-14 14:45:44,282 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23637.28 MB 2025-02-14 14:45:44,282 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 14:45:44,282 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23928.50 MB 2025-02-14 14:45:44,282 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31950.11 MB 2025-02-14 14:45:44,282 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-14 14:45:44,282 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29181.56 MB 2025-02-14 14:45:44,517 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 14:45:44,517 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 14:45:44,517 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 14:45:44,517 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:45:44,517 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25170.82 MB 2025-02-14 14:45:44,517 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25937.82 MB 2025-02-14 14:45:44,517 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 14:45:44,517 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31950.11 MB 2025-02-14 14:45:44,517 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32365.35 MB 2025-02-14 14:45:44,517 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 14:45:44,517 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26645.61 MB 2025-02-14 14:45:44,536 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 14:45:44,536 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 14:45:44,536 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:45:44,536 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:45:44,536 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26350.71 MB 2025-02-14 14:45:44,536 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26577.22 MB 2025-02-14 14:45:44,536 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 226.52 MB 2025-02-14 14:45:44,536 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32365.35 MB 2025-02-14 14:45:44,536 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32365.35 MB 2025-02-14 14:45:44,536 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:45:44,536 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26798.19 MB 2025-02-14 14:45:44,537 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 14:45:44,537 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 14:45:44,537 - resource_logging.py:150 - __exit__ - DEBUG - Time: 12.04 seconds 2025-02-14 14:45:44,537 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:45:44,537 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15111.42 MB 2025-02-14 14:45:44,537 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26777.24 MB 2025-02-14 14:45:44,537 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11665.82 MB 2025-02-14 14:45:44,537 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55377.40 MB 2025-02-14 14:45:44,537 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32365.35 MB 2025-02-14 14:45:44,537 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23012.05 MB 2025-02-14 14:45:44,537 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26798.19 MB 2025-02-14 14:45:44,801 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 14:45:44,801 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 14:45:44,801 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 14:45:44,801 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:45:44,801 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26777.24 MB 2025-02-14 14:45:44,801 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20099.42 MB 2025-02-14 14:45:44,801 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6677.81 MB 2025-02-14 14:45:44,801 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32365.35 MB 2025-02-14 14:45:44,801 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32365.35 MB 2025-02-14 14:45:44,801 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:45:44,801 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29275.70 MB 2025-02-14 14:45:44,819 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8119, cut from 8121 2025-02-14 14:45:44,819 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 14:45:44,825 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 14:45:44,825 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 14:45:44,825 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:45:44,825 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:45:44,825 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20099.42 MB 2025-02-14 14:45:44,825 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28494.64 MB 2025-02-14 14:45:44,825 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8395.21 MB 2025-02-14 14:45:44,825 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32365.35 MB 2025-02-14 14:45:44,825 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36538.68 MB 2025-02-14 14:45:44,825 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4173.33 MB 2025-02-14 14:45:44,825 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28494.64 MB 2025-02-14 14:45:44,975 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7911] 2025-02-14 14:45:44,977 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:45:44,977 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 14:45:44,978 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:45:44,978 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 14:45:44,982 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 14:45:44,983 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:45:44,983 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 14:45:44,983 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 14:47:02,041 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:47:02,042 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 14:47:02,047 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 14:47:02,052 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:47:02,053 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2493, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 14:47:02,055 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:47:02,055 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2493, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 14:47:40,514 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 14:47:40,514 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 14:47:40,514 - resource_logging.py:150 - __exit__ - DEBUG - Time: 38.44 seconds 2025-02-14 14:47:40,514 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:47:40,514 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30340.33 MB 2025-02-14 14:47:40,514 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39163.05 MB 2025-02-14 14:47:40,514 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8822.72 MB 2025-02-14 14:47:40,514 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62262.35 MB 2025-02-14 14:47:40,514 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43106.96 MB 2025-02-14 14:47:40,514 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19155.39 MB 2025-02-14 14:47:40,514 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47985.64 MB 2025-02-14 14:47:40,684 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 14:47:40,684 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 14:47:40,684 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 14:47:40,684 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:47:40,684 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39163.05 MB 2025-02-14 14:47:40,684 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28738.18 MB 2025-02-14 14:47:40,684 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10424.87 MB 2025-02-14 14:47:40,684 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43106.96 MB 2025-02-14 14:47:40,684 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 75105.30 MB 2025-02-14 14:47:40,684 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 31998.35 MB 2025-02-14 14:47:40,684 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 63698.64 MB 2025-02-14 14:47:42,660 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 14:47:42,660 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 14:47:42,660 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.97 seconds 2025-02-14 14:47:42,660 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:47:42,660 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28738.18 MB 2025-02-14 14:47:42,660 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29269.02 MB 2025-02-14 14:47:42,660 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 14:47:42,660 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 75105.30 MB 2025-02-14 14:47:42,660 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31287.41 MB 2025-02-14 14:47:42,660 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -43817.89 MB 2025-02-14 14:47:42,660 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33249.39 MB 2025-02-14 14:47:42,674 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 14:47:42,674 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 14:47:42,674 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 14:47:42,674 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:47:42,674 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29269.02 MB 2025-02-14 14:47:42,674 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31158.56 MB 2025-02-14 14:47:42,674 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 14:47:42,674 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31287.41 MB 2025-02-14 14:47:42,674 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34590.43 MB 2025-02-14 14:47:42,674 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 14:47:42,674 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32575.98 MB 2025-02-14 14:47:42,886 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 14:47:42,886 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 14:47:42,886 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 14:47:42,886 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:47:42,886 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31158.56 MB 2025-02-14 14:47:42,886 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33400.41 MB 2025-02-14 14:47:42,886 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 14:47:42,886 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34590.43 MB 2025-02-14 14:47:42,886 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41196.45 MB 2025-02-14 14:47:42,886 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 14:47:42,886 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38944.69 MB 2025-02-14 14:47:42,886 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 14:47:42,886 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 14:47:42,886 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 14:47:42,886 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:47:42,886 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29269.02 MB 2025-02-14 14:47:42,886 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33400.41 MB 2025-02-14 14:47:42,886 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 14:47:42,886 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31287.41 MB 2025-02-14 14:47:42,887 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41196.45 MB 2025-02-14 14:47:42,887 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9909.04 MB 2025-02-14 14:47:42,887 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38944.69 MB 2025-02-14 14:47:43,061 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 14:47:43,061 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 14:47:43,061 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 14:47:43,061 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:47:43,061 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34933.95 MB 2025-02-14 14:47:43,061 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35700.96 MB 2025-02-14 14:47:43,061 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 14:47:43,061 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41196.45 MB 2025-02-14 14:47:43,061 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41613.79 MB 2025-02-14 14:47:43,061 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 14:47:43,061 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36408.74 MB 2025-02-14 14:47:43,081 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 14:47:43,081 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 14:47:43,081 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:47:43,081 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:47:43,081 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36113.84 MB 2025-02-14 14:47:43,081 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36341.03 MB 2025-02-14 14:47:43,081 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.19 MB 2025-02-14 14:47:43,081 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41613.79 MB 2025-02-14 14:47:43,081 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41613.79 MB 2025-02-14 14:47:43,081 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:47:43,081 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36565.17 MB 2025-02-14 14:47:43,082 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 14:47:43,082 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 14:47:43,082 - resource_logging.py:150 - __exit__ - DEBUG - Time: 41.02 seconds 2025-02-14 14:47:43,082 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:47:43,082 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21654.52 MB 2025-02-14 14:47:43,082 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36541.19 MB 2025-02-14 14:47:43,082 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14886.68 MB 2025-02-14 14:47:43,082 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53573.84 MB 2025-02-14 14:47:43,082 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41613.79 MB 2025-02-14 14:47:43,082 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11960.06 MB 2025-02-14 14:47:43,082 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36565.17 MB 2025-02-14 14:47:43,353 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 14:47:43,353 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 14:47:43,353 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 14:47:43,353 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:47:43,353 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36541.19 MB 2025-02-14 14:47:43,353 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26644.29 MB 2025-02-14 14:47:43,353 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9896.90 MB 2025-02-14 14:47:43,353 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41613.79 MB 2025-02-14 14:47:43,353 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41613.79 MB 2025-02-14 14:47:43,353 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:47:43,353 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39041.49 MB 2025-02-14 14:47:43,371 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8125, cut from 8127 2025-02-14 14:47:43,371 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 14:47:43,377 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 14:47:43,377 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 14:47:43,377 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:47:43,377 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:47:43,377 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26644.29 MB 2025-02-14 14:47:43,377 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35044.79 MB 2025-02-14 14:47:43,377 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8400.50 MB 2025-02-14 14:47:43,377 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41613.79 MB 2025-02-14 14:47:43,377 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45791.31 MB 2025-02-14 14:47:43,377 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4177.53 MB 2025-02-14 14:47:43,377 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35044.79 MB 2025-02-14 14:47:43,542 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7917] 2025-02-14 14:47:43,543 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:47:43,543 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 14:47:43,544 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:47:43,544 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 14:47:43,549 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 14:47:43,550 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:47:43,550 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 14:47:43,550 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 14:48:43,162 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:48:43,163 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 14:48:43,168 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 14:48:43,173 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:48:43,173 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1743, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 14:48:43,174 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:48:43,174 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1743, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 14:49:10,153 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 14:49:10,153 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 14:49:10,153 - resource_logging.py:150 - __exit__ - DEBUG - Time: 26.97 seconds 2025-02-14 14:49:10,153 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:49:10,153 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25114.21 MB 2025-02-14 14:49:10,153 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31282.59 MB 2025-02-14 14:49:10,153 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6168.38 MB 2025-02-14 14:49:10,153 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54142.17 MB 2025-02-14 14:49:10,153 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39881.54 MB 2025-02-14 14:49:10,153 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14260.63 MB 2025-02-14 14:49:10,153 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40248.70 MB 2025-02-14 14:49:10,261 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 14:49:10,261 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 14:49:10,261 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 14:49:10,261 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:49:10,261 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31282.59 MB 2025-02-14 14:49:10,261 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24839.16 MB 2025-02-14 14:49:10,261 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6443.42 MB 2025-02-14 14:49:10,261 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39881.54 MB 2025-02-14 14:49:10,261 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57761.86 MB 2025-02-14 14:49:10,261 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 17880.32 MB 2025-02-14 14:49:10,261 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48879.55 MB 2025-02-14 14:49:12,195 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 14:49:12,196 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 14:49:12,196 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 14:49:12,196 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:49:12,196 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24839.16 MB 2025-02-14 14:49:12,196 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25370.01 MB 2025-02-14 14:49:12,196 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 14:49:12,196 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57761.86 MB 2025-02-14 14:49:12,196 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30949.77 MB 2025-02-14 14:49:12,196 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26812.09 MB 2025-02-14 14:49:12,196 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29349.34 MB 2025-02-14 14:49:12,212 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 14:49:12,212 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 14:49:12,212 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 14:49:12,212 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:49:12,212 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25370.01 MB 2025-02-14 14:49:12,212 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27259.54 MB 2025-02-14 14:49:12,212 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 14:49:12,212 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30949.77 MB 2025-02-14 14:49:12,212 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31893.49 MB 2025-02-14 14:49:12,212 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 14:49:12,212 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28676.97 MB 2025-02-14 14:49:12,430 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 14:49:12,430 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 14:49:12,430 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 14:49:12,430 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:49:12,430 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27259.54 MB 2025-02-14 14:49:12,430 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29501.40 MB 2025-02-14 14:49:12,430 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 14:49:12,430 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31893.49 MB 2025-02-14 14:49:12,430 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38027.66 MB 2025-02-14 14:49:12,430 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 14:49:12,430 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35045.68 MB 2025-02-14 14:49:12,430 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 14:49:12,430 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 14:49:12,431 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 14:49:12,431 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:49:12,431 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25370.01 MB 2025-02-14 14:49:12,431 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29501.40 MB 2025-02-14 14:49:12,431 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 14:49:12,431 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30949.77 MB 2025-02-14 14:49:12,431 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38027.66 MB 2025-02-14 14:49:12,431 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7077.89 MB 2025-02-14 14:49:12,431 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35045.68 MB 2025-02-14 14:49:12,606 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 14:49:12,606 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 14:49:12,606 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 14:49:12,606 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:49:12,606 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31034.94 MB 2025-02-14 14:49:12,606 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31801.94 MB 2025-02-14 14:49:12,606 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 14:49:12,606 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38027.66 MB 2025-02-14 14:49:12,606 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38444.99 MB 2025-02-14 14:49:12,606 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 14:49:12,606 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32509.73 MB 2025-02-14 14:49:12,626 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 14:49:12,626 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 14:49:12,626 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:49:12,626 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:49:12,626 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32214.83 MB 2025-02-14 14:49:12,626 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32443.22 MB 2025-02-14 14:49:12,626 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.40 MB 2025-02-14 14:49:12,626 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38444.99 MB 2025-02-14 14:49:12,626 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38444.99 MB 2025-02-14 14:49:12,626 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:49:12,626 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32667.12 MB 2025-02-14 14:49:12,627 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 14:49:12,627 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 14:49:12,627 - resource_logging.py:150 - __exit__ - DEBUG - Time: 29.45 seconds 2025-02-14 14:49:12,627 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:49:12,627 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19041.46 MB 2025-02-14 14:49:12,627 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32643.53 MB 2025-02-14 14:49:12,627 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13602.08 MB 2025-02-14 14:49:12,627 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54142.17 MB 2025-02-14 14:49:12,627 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38444.99 MB 2025-02-14 14:49:12,627 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15697.18 MB 2025-02-14 14:49:12,627 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32667.12 MB 2025-02-14 14:49:12,896 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 14:49:12,896 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 14:49:12,896 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 14:49:12,896 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:49:12,896 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32643.53 MB 2025-02-14 14:49:12,896 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24034.04 MB 2025-02-14 14:49:12,896 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8609.50 MB 2025-02-14 14:49:12,896 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38444.99 MB 2025-02-14 14:49:12,896 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38444.99 MB 2025-02-14 14:49:12,896 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:49:12,896 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35145.68 MB 2025-02-14 14:49:12,914 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8131, cut from 8133 2025-02-14 14:49:12,914 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 14:49:12,920 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 14:49:12,920 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 14:49:12,920 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:49:12,920 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:49:12,920 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24034.04 MB 2025-02-14 14:49:12,920 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32441.77 MB 2025-02-14 14:49:12,920 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8407.74 MB 2025-02-14 14:49:12,920 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38444.99 MB 2025-02-14 14:49:12,920 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42624.61 MB 2025-02-14 14:49:12,920 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4179.62 MB 2025-02-14 14:49:12,920 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32441.77 MB 2025-02-14 14:49:13,082 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7923] 2025-02-14 14:49:13,084 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:49:13,084 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 14:49:13,085 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:49:13,085 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 14:49:13,089 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 14:49:13,090 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:49:13,090 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 14:49:13,091 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 14:50:18,269 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:50:18,270 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 14:50:18,277 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 14:50:18,284 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:50:18,285 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1290, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 14:50:18,286 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:50:18,286 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1290, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 14:50:38,283 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 14:50:38,283 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 14:50:38,283 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.99 seconds 2025-02-14 14:50:38,283 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:50:38,283 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21957.63 MB 2025-02-14 14:50:38,283 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26523.13 MB 2025-02-14 14:50:38,283 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4565.50 MB 2025-02-14 14:50:38,283 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50983.86 MB 2025-02-14 14:50:38,283 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38182.85 MB 2025-02-14 14:50:38,283 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12801.02 MB 2025-02-14 14:50:38,283 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35505.87 MB 2025-02-14 14:50:38,354 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 14:50:38,354 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 14:50:38,354 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 14:50:38,354 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:50:38,354 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26523.13 MB 2025-02-14 14:50:38,354 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22484.16 MB 2025-02-14 14:50:38,354 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4038.97 MB 2025-02-14 14:50:38,354 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38182.85 MB 2025-02-14 14:50:38,354 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43989.86 MB 2025-02-14 14:50:38,354 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5807.01 MB 2025-02-14 14:50:38,354 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38544.41 MB 2025-02-14 14:50:40,272 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 14:50:40,272 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 14:50:40,272 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 14:50:40,272 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:50:40,272 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22484.16 MB 2025-02-14 14:50:40,272 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23015.00 MB 2025-02-14 14:50:40,272 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 14:50:40,272 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43989.86 MB 2025-02-14 14:50:40,272 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29441.92 MB 2025-02-14 14:50:40,272 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14547.94 MB 2025-02-14 14:50:40,272 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26994.33 MB 2025-02-14 14:50:40,286 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 14:50:40,286 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 14:50:40,286 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 14:50:40,286 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:50:40,286 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23015.00 MB 2025-02-14 14:50:40,286 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24904.53 MB 2025-02-14 14:50:40,286 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 14:50:40,286 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29441.92 MB 2025-02-14 14:50:40,286 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29441.92 MB 2025-02-14 14:50:40,286 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:50:40,286 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26321.96 MB 2025-02-14 14:50:40,493 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 14:50:40,493 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 14:50:40,493 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 14:50:40,493 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:50:40,493 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24904.53 MB 2025-02-14 14:50:40,493 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27146.39 MB 2025-02-14 14:50:40,493 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 14:50:40,493 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29441.92 MB 2025-02-14 14:50:40,493 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35104.23 MB 2025-02-14 14:50:40,493 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 14:50:40,493 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32690.67 MB 2025-02-14 14:50:40,494 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 14:50:40,494 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 14:50:40,494 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 14:50:40,494 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:50:40,494 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23015.00 MB 2025-02-14 14:50:40,494 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27146.39 MB 2025-02-14 14:50:40,494 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 14:50:40,494 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29441.92 MB 2025-02-14 14:50:40,494 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35104.23 MB 2025-02-14 14:50:40,494 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 14:50:40,494 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32690.67 MB 2025-02-14 14:50:40,663 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 14:50:40,663 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 14:50:40,663 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 14:50:40,663 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:50:40,663 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28679.93 MB 2025-02-14 14:50:40,663 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29446.93 MB 2025-02-14 14:50:40,663 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 14:50:40,663 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35104.23 MB 2025-02-14 14:50:40,663 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35521.56 MB 2025-02-14 14:50:40,663 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 14:50:40,663 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30154.72 MB 2025-02-14 14:50:40,682 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 14:50:40,682 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 14:50:40,682 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:50:40,682 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:50:40,682 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29859.82 MB 2025-02-14 14:50:40,682 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30089.28 MB 2025-02-14 14:50:40,682 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.45 MB 2025-02-14 14:50:40,682 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35521.56 MB 2025-02-14 14:50:40,682 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35521.56 MB 2025-02-14 14:50:40,682 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:50:40,682 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30291.54 MB 2025-02-14 14:50:40,683 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 14:50:40,683 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 14:50:40,683 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.39 seconds 2025-02-14 14:50:40,683 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:50:40,683 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17463.17 MB 2025-02-14 14:50:40,683 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30290.32 MB 2025-02-14 14:50:40,683 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12827.15 MB 2025-02-14 14:50:40,683 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50983.86 MB 2025-02-14 14:50:40,683 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35521.56 MB 2025-02-14 14:50:40,683 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15462.30 MB 2025-02-14 14:50:40,683 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30291.54 MB 2025-02-14 14:50:40,958 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 14:50:40,958 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 14:50:40,958 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 14:50:40,958 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:50:40,958 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30290.32 MB 2025-02-14 14:50:40,958 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22467.18 MB 2025-02-14 14:50:40,958 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7823.15 MB 2025-02-14 14:50:40,958 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35521.56 MB 2025-02-14 14:50:40,958 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35521.56 MB 2025-02-14 14:50:40,958 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:50:40,958 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32801.68 MB 2025-02-14 14:50:40,977 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8161, cut from 8163 2025-02-14 14:50:40,977 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 14:50:40,983 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 14:50:40,983 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 14:50:40,983 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:50:40,983 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:50:40,983 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22467.18 MB 2025-02-14 14:50:40,983 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30906.01 MB 2025-02-14 14:50:40,983 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8438.84 MB 2025-02-14 14:50:40,983 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35521.56 MB 2025-02-14 14:50:40,983 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43910.17 MB 2025-02-14 14:50:40,983 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8388.61 MB 2025-02-14 14:50:40,983 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30906.01 MB 2025-02-14 14:50:41,146 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7953] 2025-02-14 14:50:41,147 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:50:41,147 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 14:50:41,148 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:50:41,148 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 14:50:41,153 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 14:50:41,154 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:50:41,154 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 14:50:41,154 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 14:50:57,618 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:50:57,618 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 14:50:57,626 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 14:50:57,633 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:50:57,634 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1376, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 14:50:57,635 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:50:57,635 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1376, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 14:51:19,183 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 14:51:19,183 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 14:51:19,183 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.54 seconds 2025-02-14 14:51:19,183 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:51:19,184 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22556.89 MB 2025-02-14 14:51:19,184 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27426.48 MB 2025-02-14 14:51:19,184 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4869.59 MB 2025-02-14 14:51:19,184 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52298.78 MB 2025-02-14 14:51:19,184 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38507.91 MB 2025-02-14 14:51:19,184 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13790.87 MB 2025-02-14 14:51:19,184 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36331.62 MB 2025-02-14 14:51:19,264 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 14:51:19,264 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 14:51:19,264 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 14:51:19,264 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:51:19,264 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27426.48 MB 2025-02-14 14:51:19,264 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22931.25 MB 2025-02-14 14:51:19,264 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4495.24 MB 2025-02-14 14:51:19,264 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38507.91 MB 2025-02-14 14:51:19,264 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48060.43 MB 2025-02-14 14:51:19,264 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9552.53 MB 2025-02-14 14:51:19,264 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41849.19 MB 2025-02-14 14:51:21,205 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 14:51:21,205 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 14:51:21,205 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-14 14:51:21,205 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:51:21,205 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22931.25 MB 2025-02-14 14:51:21,205 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23462.09 MB 2025-02-14 14:51:21,205 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 14:51:21,205 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48060.43 MB 2025-02-14 14:51:21,205 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33638.32 MB 2025-02-14 14:51:21,205 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14422.11 MB 2025-02-14 14:51:21,205 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27441.42 MB 2025-02-14 14:51:21,218 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 14:51:21,218 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 14:51:21,218 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 14:51:21,218 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:51:21,218 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23462.09 MB 2025-02-14 14:51:21,218 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25351.62 MB 2025-02-14 14:51:21,218 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 14:51:21,218 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33638.32 MB 2025-02-14 14:51:21,218 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33638.32 MB 2025-02-14 14:51:21,218 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:51:21,218 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26769.05 MB 2025-02-14 14:51:21,432 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 14:51:21,432 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 14:51:21,432 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 14:51:21,432 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:51:21,432 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25351.62 MB 2025-02-14 14:51:21,432 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27593.48 MB 2025-02-14 14:51:21,432 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 14:51:21,432 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33638.32 MB 2025-02-14 14:51:21,432 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37413.19 MB 2025-02-14 14:51:21,432 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 14:51:21,432 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33137.76 MB 2025-02-14 14:51:21,433 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 14:51:21,433 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 14:51:21,433 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 14:51:21,433 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:51:21,433 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23462.09 MB 2025-02-14 14:51:21,433 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27593.48 MB 2025-02-14 14:51:21,433 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 14:51:21,433 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33638.32 MB 2025-02-14 14:51:21,433 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37413.19 MB 2025-02-14 14:51:21,433 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 14:51:21,433 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33137.76 MB 2025-02-14 14:51:21,635 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 14:51:21,635 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 14:51:21,636 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 14:51:21,636 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:51:21,636 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29127.02 MB 2025-02-14 14:51:21,636 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29894.02 MB 2025-02-14 14:51:21,636 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 14:51:21,636 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37413.19 MB 2025-02-14 14:51:21,636 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37828.43 MB 2025-02-14 14:51:21,636 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 14:51:21,636 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30601.81 MB 2025-02-14 14:51:21,663 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 14:51:21,663 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 14:51:21,663 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:51:21,663 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:51:21,663 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30306.91 MB 2025-02-14 14:51:21,663 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30534.83 MB 2025-02-14 14:51:21,663 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.92 MB 2025-02-14 14:51:21,663 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37828.43 MB 2025-02-14 14:51:21,663 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37828.43 MB 2025-02-14 14:51:21,663 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:51:21,663 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30766.23 MB 2025-02-14 14:51:21,665 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 14:51:21,665 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 14:51:21,665 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.03 seconds 2025-02-14 14:51:21,665 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:51:21,665 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17762.80 MB 2025-02-14 14:51:21,665 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30735.90 MB 2025-02-14 14:51:21,665 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12973.10 MB 2025-02-14 14:51:21,666 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52298.78 MB 2025-02-14 14:51:21,666 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37828.43 MB 2025-02-14 14:51:21,666 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14470.35 MB 2025-02-14 14:51:21,666 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30766.23 MB 2025-02-14 14:51:21,955 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 14:51:21,955 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 14:51:21,955 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.29 seconds 2025-02-14 14:51:21,955 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:51:21,955 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30735.90 MB 2025-02-14 14:51:21,955 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22767.19 MB 2025-02-14 14:51:21,955 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7968.71 MB 2025-02-14 14:51:21,955 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37828.43 MB 2025-02-14 14:51:21,955 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37828.43 MB 2025-02-14 14:51:21,955 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:51:21,955 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33247.57 MB 2025-02-14 14:51:21,974 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 14:51:21,975 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 14:51:21,982 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 14:51:21,982 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 14:51:21,982 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:51:21,982 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:51:21,982 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22767.19 MB 2025-02-14 14:51:21,982 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31206.21 MB 2025-02-14 14:51:21,982 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 14:51:21,982 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37828.43 MB 2025-02-14 14:51:21,982 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46219.13 MB 2025-02-14 14:51:21,982 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 14:51:21,982 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31206.21 MB 2025-02-14 14:51:22,255 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 14:51:22,258 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:51:22,258 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 14:51:22,260 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:51:22,260 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 14:51:22,268 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 14:51:22,270 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:51:22,270 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 14:51:22,270 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 14:52:13,146 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:52:13,147 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 14:52:13,152 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 14:52:13,156 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:52:13,156 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 287, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 14:52:13,157 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:52:13,157 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 287, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 14:52:17,606 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 14:52:17,606 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 14:52:17,606 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.44 seconds 2025-02-14 14:52:17,606 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:52:17,606 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14968.57 MB 2025-02-14 14:52:17,606 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15984.25 MB 2025-02-14 14:52:17,606 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1015.68 MB 2025-02-14 14:52:17,606 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58804.14 MB 2025-02-14 14:52:17,606 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18794.68 MB 2025-02-14 14:52:17,606 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -40009.47 MB 2025-02-14 14:52:17,606 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24893.73 MB 2025-02-14 14:52:17,627 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 14:52:17,627 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 14:52:17,627 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:52:17,627 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:52:17,627 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15984.25 MB 2025-02-14 14:52:17,628 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16329.05 MB 2025-02-14 14:52:17,628 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 344.81 MB 2025-02-14 14:52:17,628 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18794.68 MB 2025-02-14 14:52:17,628 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22536.00 MB 2025-02-14 14:52:17,628 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3741.32 MB 2025-02-14 14:52:17,628 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19774.10 MB 2025-02-14 14:52:18,907 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 14:52:18,907 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 14:52:18,907 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.28 seconds 2025-02-14 14:52:18,907 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:52:18,907 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16329.05 MB 2025-02-14 14:52:18,907 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16682.06 MB 2025-02-14 14:52:18,907 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 353.01 MB 2025-02-14 14:52:18,907 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22536.00 MB 2025-02-14 14:52:18,907 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19721.62 MB 2025-02-14 14:52:18,907 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2814.38 MB 2025-02-14 14:52:18,907 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20670.40 MB 2025-02-14 14:52:18,917 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 14:52:18,917 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 14:52:18,917 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 14:52:18,917 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:52:18,917 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16682.06 MB 2025-02-14 14:52:18,917 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17938.30 MB 2025-02-14 14:52:18,917 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1256.24 MB 2025-02-14 14:52:18,917 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19721.62 MB 2025-02-14 14:52:18,917 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20350.76 MB 2025-02-14 14:52:18,917 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 629.15 MB 2025-02-14 14:52:18,917 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18880.89 MB 2025-02-14 14:52:19,058 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 14:52:19,058 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 14:52:19,058 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-14 14:52:19,058 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:52:19,058 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17938.30 MB 2025-02-14 14:52:19,058 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19429.16 MB 2025-02-14 14:52:19,058 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1490.86 MB 2025-02-14 14:52:19,058 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20350.76 MB 2025-02-14 14:52:19,058 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24440.21 MB 2025-02-14 14:52:19,058 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4089.45 MB 2025-02-14 14:52:19,058 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23116.08 MB 2025-02-14 14:52:19,059 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 14:52:19,059 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 14:52:19,059 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 14:52:19,059 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:52:19,059 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16682.06 MB 2025-02-14 14:52:19,059 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19429.16 MB 2025-02-14 14:52:19,059 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2747.10 MB 2025-02-14 14:52:19,059 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19721.62 MB 2025-02-14 14:52:19,059 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24440.21 MB 2025-02-14 14:52:19,059 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 14:52:19,059 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23116.08 MB 2025-02-14 14:52:19,205 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 14:52:19,205 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 14:52:19,205 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-14 14:52:19,205 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:52:19,205 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20448.96 MB 2025-02-14 14:52:19,205 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20959.02 MB 2025-02-14 14:52:19,205 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 510.06 MB 2025-02-14 14:52:19,205 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24440.21 MB 2025-02-14 14:52:19,206 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24717.03 MB 2025-02-14 14:52:19,206 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 276.82 MB 2025-02-14 14:52:19,206 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21429.70 MB 2025-02-14 14:52:19,220 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 14:52:19,220 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 14:52:19,220 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 14:52:19,220 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:52:19,220 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21233.59 MB 2025-02-14 14:52:19,220 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21438.32 MB 2025-02-14 14:52:19,220 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 204.73 MB 2025-02-14 14:52:19,220 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24717.03 MB 2025-02-14 14:52:19,220 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24721.23 MB 2025-02-14 14:52:19,220 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-14 14:52:19,220 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21502.06 MB 2025-02-14 14:52:19,221 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 14:52:19,221 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 14:52:19,221 - resource_logging.py:150 - __exit__ - DEBUG - Time: 6.06 seconds 2025-02-14 14:52:19,221 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:52:19,221 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13968.64 MB 2025-02-14 14:52:19,221 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21639.40 MB 2025-02-14 14:52:19,221 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7670.76 MB 2025-02-14 14:52:19,221 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58804.14 MB 2025-02-14 14:52:19,221 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24721.23 MB 2025-02-14 14:52:19,221 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34082.91 MB 2025-02-14 14:52:19,221 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21639.40 MB 2025-02-14 14:52:19,492 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 14:52:19,492 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 14:52:19,492 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 14:52:19,492 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:52:19,492 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21639.40 MB 2025-02-14 14:52:19,492 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24653.43 MB 2025-02-14 14:52:19,492 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-14 14:52:19,492 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24721.23 MB 2025-02-14 14:52:19,492 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26197.62 MB 2025-02-14 14:52:19,492 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1476.40 MB 2025-02-14 14:52:19,492 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24955.06 MB 2025-02-14 14:52:19,510 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 14:52:19,510 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 14:52:19,516 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 14:52:19,516 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 14:52:19,516 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:52:19,516 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:52:19,516 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18340.12 MB 2025-02-14 14:52:19,516 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26779.14 MB 2025-02-14 14:52:19,516 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 14:52:19,516 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26197.62 MB 2025-02-14 14:52:19,516 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36687.58 MB 2025-02-14 14:52:19,516 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 14:52:19,516 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26779.14 MB 2025-02-14 14:52:19,679 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 14:52:19,680 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:52:19,680 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 14:52:19,681 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:52:19,681 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 14:52:19,686 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 14:52:19,687 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:52:19,687 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 14:52:19,687 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 14:52:30,024 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:52:30,024 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 14:52:30,029 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 14:52:30,032 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:52:30,032 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1130, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 14:52:30,033 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:52:30,033 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1130, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 14:52:47,580 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 14:52:47,580 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 14:52:47,580 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.54 seconds 2025-02-14 14:52:47,580 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:52:47,580 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20842.73 MB 2025-02-14 14:52:47,580 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24842.00 MB 2025-02-14 14:52:47,580 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3999.27 MB 2025-02-14 14:52:47,580 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49272.59 MB 2025-02-14 14:52:47,580 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31396.46 MB 2025-02-14 14:52:47,580 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17876.12 MB 2025-02-14 14:52:47,580 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33711.48 MB 2025-02-14 14:52:47,653 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 14:52:47,653 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 14:52:47,653 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 14:52:47,653 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:52:47,653 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24842.00 MB 2025-02-14 14:52:47,653 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21652.37 MB 2025-02-14 14:52:47,653 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3189.63 MB 2025-02-14 14:52:47,653 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31396.46 MB 2025-02-14 14:52:47,653 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42324.72 MB 2025-02-14 14:52:47,653 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10928.26 MB 2025-02-14 14:52:47,653 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36990.01 MB 2025-02-14 14:52:49,585 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 14:52:49,585 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 14:52:49,585 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 14:52:49,586 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:52:49,586 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21652.37 MB 2025-02-14 14:52:49,586 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22183.21 MB 2025-02-14 14:52:49,586 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 14:52:49,586 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42324.72 MB 2025-02-14 14:52:49,586 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28812.77 MB 2025-02-14 14:52:49,586 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13511.95 MB 2025-02-14 14:52:49,586 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26162.54 MB 2025-02-14 14:52:49,599 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 14:52:49,599 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 14:52:49,599 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 14:52:49,599 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:52:49,599 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22183.21 MB 2025-02-14 14:52:49,599 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24072.74 MB 2025-02-14 14:52:49,599 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 14:52:49,599 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28812.77 MB 2025-02-14 14:52:49,599 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28812.77 MB 2025-02-14 14:52:49,599 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:52:49,599 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25490.17 MB 2025-02-14 14:52:49,813 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 14:52:49,813 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 14:52:49,813 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 14:52:49,813 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:52:49,813 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24072.74 MB 2025-02-14 14:52:49,813 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26314.60 MB 2025-02-14 14:52:49,813 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 14:52:49,813 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28812.77 MB 2025-02-14 14:52:49,813 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34475.08 MB 2025-02-14 14:52:49,813 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 14:52:49,813 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31858.88 MB 2025-02-14 14:52:49,814 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 14:52:49,814 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 14:52:49,814 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 14:52:49,814 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:52:49,814 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22183.21 MB 2025-02-14 14:52:49,814 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26314.60 MB 2025-02-14 14:52:49,814 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 14:52:49,814 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28812.77 MB 2025-02-14 14:52:49,814 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34475.08 MB 2025-02-14 14:52:49,814 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 14:52:49,814 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31858.88 MB 2025-02-14 14:52:49,986 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 14:52:49,986 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 14:52:49,986 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 14:52:49,986 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:52:49,986 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27848.14 MB 2025-02-14 14:52:49,986 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28615.14 MB 2025-02-14 14:52:49,986 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 14:52:49,986 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34475.08 MB 2025-02-14 14:52:49,986 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34890.32 MB 2025-02-14 14:52:49,986 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 14:52:49,986 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29322.93 MB 2025-02-14 14:52:50,005 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 14:52:50,005 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 14:52:50,005 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:52:50,005 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:52:50,005 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29028.03 MB 2025-02-14 14:52:50,005 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29256.05 MB 2025-02-14 14:52:50,005 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.02 MB 2025-02-14 14:52:50,005 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34890.32 MB 2025-02-14 14:52:50,005 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34890.32 MB 2025-02-14 14:52:50,005 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:52:50,005 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29490.83 MB 2025-02-14 14:52:50,006 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 14:52:50,006 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 14:52:50,006 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.97 seconds 2025-02-14 14:52:50,006 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:52:50,006 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16905.72 MB 2025-02-14 14:52:50,006 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29457.12 MB 2025-02-14 14:52:50,006 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12551.40 MB 2025-02-14 14:52:50,007 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49272.59 MB 2025-02-14 14:52:50,007 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34890.32 MB 2025-02-14 14:52:50,007 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14382.27 MB 2025-02-14 14:52:50,007 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29490.83 MB 2025-02-14 14:52:50,277 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 14:52:50,277 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 14:52:50,277 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 14:52:50,277 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:52:50,277 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29457.12 MB 2025-02-14 14:52:50,277 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21910.11 MB 2025-02-14 14:52:50,277 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7547.01 MB 2025-02-14 14:52:50,277 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34890.32 MB 2025-02-14 14:52:50,277 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34890.32 MB 2025-02-14 14:52:50,277 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:52:50,277 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31968.79 MB 2025-02-14 14:52:50,295 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 14:52:50,295 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 14:52:50,302 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 14:52:50,302 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 14:52:50,302 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:52:50,302 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:52:50,302 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21910.11 MB 2025-02-14 14:52:50,302 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30349.13 MB 2025-02-14 14:52:50,302 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 14:52:50,302 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34890.32 MB 2025-02-14 14:52:50,302 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43281.02 MB 2025-02-14 14:52:50,302 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 14:52:50,302 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30349.13 MB 2025-02-14 14:52:50,465 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 14:52:50,467 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:52:50,467 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 14:52:50,467 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:52:50,468 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 14:52:50,472 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 14:52:50,473 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:52:50,473 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 14:52:50,473 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 14:53:02,774 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:53:02,774 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 14:53:02,779 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 14:53:02,782 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:53:02,783 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 154, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 14:53:02,783 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:53:02,783 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 154, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 14:53:05,211 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 14:53:05,211 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 14:53:05,211 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.42 seconds 2025-02-14 14:53:05,211 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:53:05,211 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14041.80 MB 2025-02-14 14:53:05,211 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14586.80 MB 2025-02-14 14:53:05,211 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 545.00 MB 2025-02-14 14:53:05,211 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55866.03 MB 2025-02-14 14:53:05,211 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17855.15 MB 2025-02-14 14:53:05,211 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -38010.88 MB 2025-02-14 14:53:05,211 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23513.98 MB 2025-02-14 14:53:05,222 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 14:53:05,222 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 14:53:05,222 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 14:53:05,222 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:53:05,222 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14586.80 MB 2025-02-14 14:53:05,222 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14710.39 MB 2025-02-14 14:53:05,222 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 123.59 MB 2025-02-14 14:53:05,222 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17855.15 MB 2025-02-14 14:53:05,222 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17855.15 MB 2025-02-14 14:53:05,222 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:53:05,222 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16504.43 MB 2025-02-14 14:53:05,893 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 14:53:05,893 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 14:53:05,893 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.67 seconds 2025-02-14 14:53:05,893 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:53:05,893 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14710.39 MB 2025-02-14 14:53:05,893 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14888.22 MB 2025-02-14 14:53:05,893 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 177.83 MB 2025-02-14 14:53:05,893 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17855.15 MB 2025-02-14 14:53:05,893 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17383.29 MB 2025-02-14 14:53:05,893 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -471.86 MB 2025-02-14 14:53:05,893 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18881.86 MB 2025-02-14 14:53:05,899 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 14:53:05,899 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 14:53:05,899 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 14:53:05,899 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:53:05,900 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14888.16 MB 2025-02-14 14:53:05,900 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15521.00 MB 2025-02-14 14:53:05,900 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 632.84 MB 2025-02-14 14:53:05,900 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17383.29 MB 2025-02-14 14:53:05,900 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17383.29 MB 2025-02-14 14:53:05,900 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:53:05,900 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 15995.84 MB 2025-02-14 14:53:05,974 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 14:53:05,974 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 14:53:05,974 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 14:53:05,974 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:53:05,974 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15521.00 MB 2025-02-14 14:53:05,974 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16272.06 MB 2025-02-14 14:53:05,974 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 751.06 MB 2025-02-14 14:53:05,974 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17383.29 MB 2025-02-14 14:53:05,974 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19126.03 MB 2025-02-14 14:53:05,974 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1742.73 MB 2025-02-14 14:53:05,974 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18131.45 MB 2025-02-14 14:53:05,975 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 14:53:05,975 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 14:53:05,975 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 14:53:05,975 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:53:05,975 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14888.16 MB 2025-02-14 14:53:05,975 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16272.06 MB 2025-02-14 14:53:05,975 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1383.90 MB 2025-02-14 14:53:05,975 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17383.29 MB 2025-02-14 14:53:05,975 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19126.03 MB 2025-02-14 14:53:05,975 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1742.73 MB 2025-02-14 14:53:05,975 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18131.45 MB 2025-02-14 14:53:06,033 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 14:53:06,033 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 14:53:06,033 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 14:53:06,033 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:53:06,033 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16785.80 MB 2025-02-14 14:53:06,033 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17042.74 MB 2025-02-14 14:53:06,033 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 256.95 MB 2025-02-14 14:53:06,033 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19126.03 MB 2025-02-14 14:53:06,033 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19260.24 MB 2025-02-14 14:53:06,033 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 134.22 MB 2025-02-14 14:53:06,033 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17291.94 MB 2025-02-14 14:53:06,041 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 14:53:06,041 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 14:53:06,041 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 14:53:06,041 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:53:06,041 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17181.07 MB 2025-02-14 14:53:06,041 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17401.74 MB 2025-02-14 14:53:06,041 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 220.67 MB 2025-02-14 14:53:06,041 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19260.24 MB 2025-02-14 14:53:06,041 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19260.24 MB 2025-02-14 14:53:06,041 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:53:06,041 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17401.74 MB 2025-02-14 14:53:06,043 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 14:53:06,043 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 14:53:06,043 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.26 seconds 2025-02-14 14:53:06,043 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:53:06,043 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13505.25 MB 2025-02-14 14:53:06,043 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14240.18 MB 2025-02-14 14:53:06,043 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 734.93 MB 2025-02-14 14:53:06,043 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55866.03 MB 2025-02-14 14:53:06,043 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19260.24 MB 2025-02-14 14:53:06,043 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -36605.79 MB 2025-02-14 14:53:06,043 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17602.61 MB 2025-02-14 14:53:06,310 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 14:53:06,310 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 14:53:06,310 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 14:53:06,310 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:53:06,310 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14240.18 MB 2025-02-14 14:53:06,310 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17251.26 MB 2025-02-14 14:53:06,310 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3011.08 MB 2025-02-14 14:53:06,310 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19260.24 MB 2025-02-14 14:53:06,310 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19260.24 MB 2025-02-14 14:53:06,310 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:53:06,310 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17552.34 MB 2025-02-14 14:53:06,340 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8154, cut from 8156 2025-02-14 14:53:06,340 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 14:53:06,348 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 14:53:06,348 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 14:53:06,348 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 14:53:06,348 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:53:06,348 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17251.26 MB 2025-02-14 14:53:06,348 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25681.94 MB 2025-02-14 14:53:06,348 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8430.68 MB 2025-02-14 14:53:06,348 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19260.24 MB 2025-02-14 14:53:06,348 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29739.71 MB 2025-02-14 14:53:06,348 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10479.47 MB 2025-02-14 14:53:06,348 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25681.94 MB 2025-02-14 14:53:06,516 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7946] 2025-02-14 14:53:06,517 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:53:06,517 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 14:53:06,518 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:53:06,518 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 14:53:06,523 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 14:53:06,524 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:53:06,524 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 14:53:06,524 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 14:54:29,773 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:54:29,773 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 14:54:29,778 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 14:54:29,783 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:54:29,783 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 228, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 14:54:29,784 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:54:29,784 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 228, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 14:54:33,309 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 14:54:33,309 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 14:54:33,309 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.52 seconds 2025-02-14 14:54:33,309 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:54:33,309 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14557.45 MB 2025-02-14 14:54:33,309 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15364.33 MB 2025-02-14 14:54:33,309 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 806.88 MB 2025-02-14 14:54:33,309 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42312.14 MB 2025-02-14 14:54:33,309 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17381.20 MB 2025-02-14 14:54:33,309 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24930.94 MB 2025-02-14 14:54:33,309 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24256.12 MB 2025-02-14 14:54:33,325 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 14:54:33,325 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 14:54:33,325 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 14:54:33,325 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:54:33,325 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15364.33 MB 2025-02-14 14:54:33,325 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15390.28 MB 2025-02-14 14:54:33,325 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 25.95 MB 2025-02-14 14:54:33,325 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17381.20 MB 2025-02-14 14:54:33,325 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18928.89 MB 2025-02-14 14:54:33,325 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1547.70 MB 2025-02-14 14:54:33,325 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17850.87 MB 2025-02-14 14:54:34,168 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 14:54:34,168 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 14:54:34,168 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.84 seconds 2025-02-14 14:54:34,168 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:54:34,168 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15390.28 MB 2025-02-14 14:54:34,168 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15623.85 MB 2025-02-14 14:54:34,168 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 233.57 MB 2025-02-14 14:54:34,168 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18928.89 MB 2025-02-14 14:54:34,168 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18608.03 MB 2025-02-14 14:54:34,168 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -320.86 MB 2025-02-14 14:54:34,168 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19561.75 MB 2025-02-14 14:54:34,176 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 14:54:34,176 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 14:54:34,177 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 14:54:34,177 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:54:34,177 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15623.78 MB 2025-02-14 14:54:34,177 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16454.98 MB 2025-02-14 14:54:34,177 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 831.19 MB 2025-02-14 14:54:34,177 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18608.03 MB 2025-02-14 14:54:34,177 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18608.03 MB 2025-02-14 14:54:34,177 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:54:34,177 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17078.65 MB 2025-02-14 14:54:34,272 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 14:54:34,273 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 14:54:34,273 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 14:54:34,273 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:54:34,273 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16454.98 MB 2025-02-14 14:54:34,273 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17441.43 MB 2025-02-14 14:54:34,273 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 986.45 MB 2025-02-14 14:54:34,273 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18608.03 MB 2025-02-14 14:54:34,273 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20891.83 MB 2025-02-14 14:54:34,273 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2283.80 MB 2025-02-14 14:54:34,273 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19880.88 MB 2025-02-14 14:54:34,273 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 14:54:34,273 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 14:54:34,273 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 14:54:34,273 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:54:34,273 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15623.78 MB 2025-02-14 14:54:34,273 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17441.43 MB 2025-02-14 14:54:34,273 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1817.65 MB 2025-02-14 14:54:34,273 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18608.03 MB 2025-02-14 14:54:34,273 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20891.83 MB 2025-02-14 14:54:34,273 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2283.80 MB 2025-02-14 14:54:34,273 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19880.88 MB 2025-02-14 14:54:34,346 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 14:54:34,347 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 14:54:34,347 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 14:54:34,347 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:54:34,347 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18116.19 MB 2025-02-14 14:54:34,347 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18453.67 MB 2025-02-14 14:54:34,347 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 337.48 MB 2025-02-14 14:54:34,347 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20891.83 MB 2025-02-14 14:54:34,347 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21072.18 MB 2025-02-14 14:54:34,347 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 180.36 MB 2025-02-14 14:54:34,347 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18771.77 MB 2025-02-14 14:54:34,356 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 14:54:34,356 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 14:54:34,356 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 14:54:34,357 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:54:34,357 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18635.35 MB 2025-02-14 14:54:34,357 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18862.87 MB 2025-02-14 14:54:34,357 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.52 MB 2025-02-14 14:54:34,357 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21072.18 MB 2025-02-14 14:54:34,357 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21072.18 MB 2025-02-14 14:54:34,357 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:54:34,357 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18886.42 MB 2025-02-14 14:54:34,358 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 14:54:34,358 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 14:54:34,358 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.57 seconds 2025-02-14 14:54:34,358 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:54:34,358 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13763.08 MB 2025-02-14 14:54:34,358 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19063.77 MB 2025-02-14 14:54:34,358 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5300.69 MB 2025-02-14 14:54:34,358 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42312.14 MB 2025-02-14 14:54:34,358 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21072.18 MB 2025-02-14 14:54:34,358 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21239.96 MB 2025-02-14 14:54:34,358 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19063.77 MB 2025-02-14 14:54:34,623 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 14:54:34,623 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 14:54:34,623 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 14:54:34,623 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:54:34,623 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19063.77 MB 2025-02-14 14:54:34,623 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17709.72 MB 2025-02-14 14:54:34,623 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1354.05 MB 2025-02-14 14:54:34,623 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21072.18 MB 2025-02-14 14:54:34,623 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21072.18 MB 2025-02-14 14:54:34,623 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:54:34,623 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19297.99 MB 2025-02-14 14:54:34,641 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8155, cut from 8157 2025-02-14 14:54:34,642 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 14:54:34,648 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 14:54:34,648 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 14:54:34,648 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:54:34,648 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:54:34,648 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17709.72 MB 2025-02-14 14:54:34,648 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26141.18 MB 2025-02-14 14:54:34,648 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8431.46 MB 2025-02-14 14:54:34,648 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21072.18 MB 2025-02-14 14:54:34,648 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31553.75 MB 2025-02-14 14:54:34,648 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10481.57 MB 2025-02-14 14:54:34,648 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26141.18 MB 2025-02-14 14:54:34,799 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7947] 2025-02-14 14:54:34,801 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:54:34,801 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 14:54:34,802 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:54:34,802 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 14:54:34,806 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 14:54:34,807 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:54:34,808 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 14:54:34,808 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 14:54:54,645 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:54:54,645 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 14:54:54,649 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 14:54:54,653 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:54:54,653 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1832, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 14:54:54,654 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:54:54,654 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1832, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 14:55:22,976 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 14:55:22,976 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 14:55:22,976 - resource_logging.py:150 - __exit__ - DEBUG - Time: 28.31 seconds 2025-02-14 14:55:22,977 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:55:22,977 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25734.38 MB 2025-02-14 14:55:22,977 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32218.77 MB 2025-02-14 14:55:22,977 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6484.39 MB 2025-02-14 14:55:22,977 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39938.16 MB 2025-02-14 14:55:22,977 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40162.56 MB 2025-02-14 14:55:22,977 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 224.40 MB 2025-02-14 14:55:22,977 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41095.36 MB 2025-02-14 14:55:23,092 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 14:55:23,092 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 14:55:23,092 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 14:55:23,092 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:55:23,092 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32218.77 MB 2025-02-14 14:55:23,092 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25301.85 MB 2025-02-14 14:55:23,092 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6916.92 MB 2025-02-14 14:55:23,092 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40162.56 MB 2025-02-14 14:55:23,092 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 60242.79 MB 2025-02-14 14:55:23,092 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 20080.23 MB 2025-02-14 14:55:23,092 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50944.46 MB 2025-02-14 14:55:25,035 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 14:55:25,035 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 14:55:25,035 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-14 14:55:25,035 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:55:25,035 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25301.85 MB 2025-02-14 14:55:25,035 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25832.69 MB 2025-02-14 14:55:25,036 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 14:55:25,036 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60242.79 MB 2025-02-14 14:55:25,036 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30901.53 MB 2025-02-14 14:55:25,036 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29341.25 MB 2025-02-14 14:55:25,036 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29812.02 MB 2025-02-14 14:55:25,052 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 14:55:25,052 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 14:55:25,052 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 14:55:25,052 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:55:25,052 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25832.69 MB 2025-02-14 14:55:25,052 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27722.22 MB 2025-02-14 14:55:25,052 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 14:55:25,052 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30901.53 MB 2025-02-14 14:55:25,052 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31845.25 MB 2025-02-14 14:55:25,052 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 14:55:25,052 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29139.65 MB 2025-02-14 14:55:25,264 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 14:55:25,264 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 14:55:25,264 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 14:55:25,264 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:55:25,264 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27722.22 MB 2025-02-14 14:55:25,264 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29964.08 MB 2025-02-14 14:55:25,264 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 14:55:25,264 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31845.25 MB 2025-02-14 14:55:25,264 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37979.42 MB 2025-02-14 14:55:25,264 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 14:55:25,264 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35508.36 MB 2025-02-14 14:55:25,265 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 14:55:25,265 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 14:55:25,265 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 14:55:25,265 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:55:25,265 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25832.69 MB 2025-02-14 14:55:25,265 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29964.08 MB 2025-02-14 14:55:25,265 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 14:55:25,265 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30901.53 MB 2025-02-14 14:55:25,265 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37979.42 MB 2025-02-14 14:55:25,265 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7077.89 MB 2025-02-14 14:55:25,265 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35508.36 MB 2025-02-14 14:55:25,438 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 14:55:25,438 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 14:55:25,438 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 14:55:25,438 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:55:25,438 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31497.62 MB 2025-02-14 14:55:25,438 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32264.62 MB 2025-02-14 14:55:25,438 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 14:55:25,438 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37979.42 MB 2025-02-14 14:55:25,438 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38394.66 MB 2025-02-14 14:55:25,438 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 14:55:25,438 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32972.41 MB 2025-02-14 14:55:25,457 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 14:55:25,457 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 14:55:25,457 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:55:25,457 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:55:25,457 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32677.51 MB 2025-02-14 14:55:25,457 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32907.00 MB 2025-02-14 14:55:25,457 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.49 MB 2025-02-14 14:55:25,457 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38394.66 MB 2025-02-14 14:55:25,457 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38394.66 MB 2025-02-14 14:55:25,457 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:55:25,457 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33119.04 MB 2025-02-14 14:55:25,458 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 14:55:25,458 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 14:55:25,458 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.80 seconds 2025-02-14 14:55:25,458 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:55:25,459 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19351.54 MB 2025-02-14 14:55:25,459 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33107.51 MB 2025-02-14 14:55:25,459 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13755.97 MB 2025-02-14 14:55:25,459 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39938.16 MB 2025-02-14 14:55:25,459 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38394.66 MB 2025-02-14 14:55:25,459 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1543.50 MB 2025-02-14 14:55:25,459 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33119.04 MB 2025-02-14 14:55:25,732 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 14:55:25,732 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 14:55:25,732 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 14:55:25,732 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:55:25,732 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33107.51 MB 2025-02-14 14:55:25,732 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24347.17 MB 2025-02-14 14:55:25,732 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8760.34 MB 2025-02-14 14:55:25,732 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38394.66 MB 2025-02-14 14:55:25,732 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38394.66 MB 2025-02-14 14:55:25,732 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:55:25,732 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35612.11 MB 2025-02-14 14:55:25,750 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8139, cut from 8141 2025-02-14 14:55:25,750 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 14:55:25,756 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 14:55:25,756 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 14:55:25,756 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:55:25,756 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:55:25,756 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24347.17 MB 2025-02-14 14:55:25,756 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32762.12 MB 2025-02-14 14:55:25,756 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8414.95 MB 2025-02-14 14:55:25,756 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38394.66 MB 2025-02-14 14:55:25,756 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42578.48 MB 2025-02-14 14:55:25,756 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4183.82 MB 2025-02-14 14:55:25,756 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32762.12 MB 2025-02-14 14:55:25,923 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7931] 2025-02-14 14:55:25,924 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:55:25,924 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 14:55:25,925 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:55:25,925 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 14:55:25,930 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 14:55:25,931 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:55:25,931 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 14:55:25,931 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 14:56:23,966 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:56:23,966 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 14:56:23,971 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 14:56:23,975 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:56:23,975 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 462, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 14:56:23,976 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:56:23,976 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 462, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 14:56:31,129 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 14:56:31,129 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 14:56:31,129 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.15 seconds 2025-02-14 14:56:31,129 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:56:31,129 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16188.00 MB 2025-02-14 14:56:31,129 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17823.77 MB 2025-02-14 14:56:31,129 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1635.78 MB 2025-02-14 14:56:31,129 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50946.11 MB 2025-02-14 14:56:31,129 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22319.99 MB 2025-02-14 14:56:31,129 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -28626.12 MB 2025-02-14 14:56:31,129 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26792.64 MB 2025-02-14 14:56:31,165 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 14:56:31,165 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 14:56:31,165 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 14:56:31,165 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:56:31,165 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17823.77 MB 2025-02-14 14:56:31,165 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18180.69 MB 2025-02-14 14:56:31,165 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 356.92 MB 2025-02-14 14:56:31,165 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22319.99 MB 2025-02-14 14:56:31,165 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28221.37 MB 2025-02-14 14:56:31,165 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5901.39 MB 2025-02-14 14:56:31,165 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25253.15 MB 2025-02-14 14:56:33,075 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 14:56:33,075 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 14:56:33,075 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 14:56:33,075 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:56:33,075 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18180.69 MB 2025-02-14 14:56:33,075 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18711.53 MB 2025-02-14 14:56:33,075 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 14:56:33,075 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28221.37 MB 2025-02-14 14:56:33,075 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21864.91 MB 2025-02-14 14:56:33,075 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6356.47 MB 2025-02-14 14:56:33,075 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22691.91 MB 2025-02-14 14:56:33,089 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 14:56:33,089 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 14:56:33,089 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 14:56:33,089 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:56:33,089 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18711.53 MB 2025-02-14 14:56:33,089 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20601.07 MB 2025-02-14 14:56:33,089 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 14:56:33,089 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21864.91 MB 2025-02-14 14:56:33,089 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24696.06 MB 2025-02-14 14:56:33,089 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-14 14:56:33,089 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22018.50 MB 2025-02-14 14:56:33,293 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 14:56:33,293 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 14:56:33,294 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 14:56:33,294 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:56:33,294 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20601.07 MB 2025-02-14 14:56:33,294 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22842.92 MB 2025-02-14 14:56:33,294 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 14:56:33,294 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24696.06 MB 2025-02-14 14:56:33,294 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30830.23 MB 2025-02-14 14:56:33,294 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 14:56:33,294 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28387.21 MB 2025-02-14 14:56:33,294 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 14:56:33,294 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 14:56:33,294 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 14:56:33,294 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:56:33,294 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18711.53 MB 2025-02-14 14:56:33,294 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22842.92 MB 2025-02-14 14:56:33,294 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 14:56:33,294 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21864.91 MB 2025-02-14 14:56:33,294 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30830.23 MB 2025-02-14 14:56:33,294 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8965.32 MB 2025-02-14 14:56:33,294 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28387.21 MB 2025-02-14 14:56:33,457 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 14:56:33,457 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 14:56:33,457 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 14:56:33,457 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:56:33,457 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24376.47 MB 2025-02-14 14:56:33,457 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25143.47 MB 2025-02-14 14:56:33,457 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 14:56:33,457 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30830.23 MB 2025-02-14 14:56:33,457 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31245.47 MB 2025-02-14 14:56:33,457 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 14:56:33,457 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25851.26 MB 2025-02-14 14:56:33,475 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 14:56:33,476 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 14:56:33,476 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:56:33,476 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:56:33,476 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25556.36 MB 2025-02-14 14:56:33,476 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25784.08 MB 2025-02-14 14:56:33,476 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.72 MB 2025-02-14 14:56:33,476 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31245.47 MB 2025-02-14 14:56:33,476 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31245.47 MB 2025-02-14 14:56:33,476 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:56:33,476 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25988.46 MB 2025-02-14 14:56:33,477 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 14:56:33,477 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 14:56:33,477 - resource_logging.py:150 - __exit__ - DEBUG - Time: 9.50 seconds 2025-02-14 14:56:33,477 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:56:33,477 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14578.35 MB 2025-02-14 14:56:33,477 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25985.15 MB 2025-02-14 14:56:33,477 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11406.80 MB 2025-02-14 14:56:33,477 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50946.11 MB 2025-02-14 14:56:33,477 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31245.47 MB 2025-02-14 14:56:33,477 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19700.65 MB 2025-02-14 14:56:33,477 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25988.46 MB 2025-02-14 14:56:33,745 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 14:56:33,745 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 14:56:33,745 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 14:56:33,745 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:56:33,745 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25985.15 MB 2025-02-14 14:56:33,745 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19582.74 MB 2025-02-14 14:56:33,745 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6402.41 MB 2025-02-14 14:56:33,745 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31245.47 MB 2025-02-14 14:56:33,745 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31245.47 MB 2025-02-14 14:56:33,745 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:56:33,745 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28496.82 MB 2025-02-14 14:56:33,763 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 14:56:33,763 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-14 14:56:33,769 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 14:56:33,769 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 14:56:33,769 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:56:33,769 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:56:33,769 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19582.74 MB 2025-02-14 14:56:33,769 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28021.76 MB 2025-02-14 14:56:33,769 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 14:56:33,769 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31245.47 MB 2025-02-14 14:56:33,769 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41735.42 MB 2025-02-14 14:56:33,769 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 14:56:33,769 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28021.76 MB 2025-02-14 14:56:33,920 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 14:56:33,922 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:56:33,922 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 14:56:33,923 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:56:33,923 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 14:56:33,927 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 14:56:33,928 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:56:33,928 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 14:56:33,928 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-14 14:57:57,358 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:57:57,358 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 14:57:57,364 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 14:57:57,367 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:57:57,368 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1248, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 14:57:57,368 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:57:57,369 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1248, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 14:58:16,518 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 14:58:16,518 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 14:58:16,518 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.14 seconds 2025-02-14 14:58:16,518 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:58:16,518 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21664.97 MB 2025-02-14 14:58:16,518 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26081.57 MB 2025-02-14 14:58:16,518 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4416.60 MB 2025-02-14 14:58:16,518 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54320.43 MB 2025-02-14 14:58:16,518 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38105.25 MB 2025-02-14 14:58:16,518 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16215.18 MB 2025-02-14 14:58:16,518 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34986.71 MB 2025-02-14 14:58:16,593 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 14:58:16,593 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 14:58:16,593 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 14:58:16,593 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:58:16,593 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26081.57 MB 2025-02-14 14:58:16,593 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22265.81 MB 2025-02-14 14:58:16,593 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3815.76 MB 2025-02-14 14:58:16,593 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38105.25 MB 2025-02-14 14:58:16,593 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46795.85 MB 2025-02-14 14:58:16,593 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8690.60 MB 2025-02-14 14:58:16,593 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39204.77 MB 2025-02-14 14:58:18,505 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 14:58:18,506 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 14:58:18,506 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 14:58:18,506 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:58:18,506 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22265.81 MB 2025-02-14 14:58:18,506 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22796.65 MB 2025-02-14 14:58:18,506 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 14:58:18,506 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46795.85 MB 2025-02-14 14:58:18,506 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33688.65 MB 2025-02-14 14:58:18,506 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13107.20 MB 2025-02-14 14:58:18,506 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26775.99 MB 2025-02-14 14:58:18,519 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 14:58:18,519 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 14:58:18,519 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 14:58:18,519 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:58:18,519 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22796.65 MB 2025-02-14 14:58:18,519 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24686.19 MB 2025-02-14 14:58:18,519 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 14:58:18,519 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33688.65 MB 2025-02-14 14:58:18,519 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33688.65 MB 2025-02-14 14:58:18,519 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:58:18,519 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26103.62 MB 2025-02-14 14:58:18,733 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 14:58:18,733 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 14:58:18,733 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 14:58:18,733 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:58:18,733 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24686.19 MB 2025-02-14 14:58:18,733 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26928.04 MB 2025-02-14 14:58:18,733 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 14:58:18,733 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33688.65 MB 2025-02-14 14:58:18,733 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35576.09 MB 2025-02-14 14:58:18,733 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 14:58:18,733 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32472.33 MB 2025-02-14 14:58:18,734 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 14:58:18,734 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 14:58:18,734 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 14:58:18,734 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:58:18,734 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22796.65 MB 2025-02-14 14:58:18,734 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26928.04 MB 2025-02-14 14:58:18,734 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 14:58:18,734 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33688.65 MB 2025-02-14 14:58:18,734 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35576.09 MB 2025-02-14 14:58:18,734 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 14:58:18,734 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32472.33 MB 2025-02-14 14:58:18,906 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 14:58:18,906 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 14:58:18,906 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 14:58:18,906 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:58:18,906 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28461.59 MB 2025-02-14 14:58:18,906 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29228.59 MB 2025-02-14 14:58:18,906 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 14:58:18,906 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35576.09 MB 2025-02-14 14:58:18,906 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35991.32 MB 2025-02-14 14:58:18,906 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 14:58:18,906 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29936.38 MB 2025-02-14 14:58:18,926 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 14:58:18,926 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 14:58:18,926 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:58:18,926 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:58:18,926 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29641.48 MB 2025-02-14 14:58:18,926 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29869.80 MB 2025-02-14 14:58:18,926 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.32 MB 2025-02-14 14:58:18,926 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35991.32 MB 2025-02-14 14:58:18,926 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35991.32 MB 2025-02-14 14:58:18,926 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:58:18,926 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30105.63 MB 2025-02-14 14:58:18,927 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 14:58:18,927 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 14:58:18,927 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.56 seconds 2025-02-14 14:58:18,927 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:58:18,927 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17316.84 MB 2025-02-14 14:58:18,927 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30070.77 MB 2025-02-14 14:58:18,927 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12753.94 MB 2025-02-14 14:58:18,927 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54320.43 MB 2025-02-14 14:58:18,927 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35991.32 MB 2025-02-14 14:58:18,927 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18329.11 MB 2025-02-14 14:58:18,927 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30105.63 MB 2025-02-14 14:58:19,195 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 14:58:19,195 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 14:58:19,195 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 14:58:19,195 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:58:19,195 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30070.77 MB 2025-02-14 14:58:19,195 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22319.90 MB 2025-02-14 14:58:19,195 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7750.87 MB 2025-02-14 14:58:19,195 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35991.32 MB 2025-02-14 14:58:19,195 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35991.32 MB 2025-02-14 14:58:19,195 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:58:19,195 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32581.21 MB 2025-02-14 14:58:19,212 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8158, cut from 8160 2025-02-14 14:58:19,212 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 14:58:19,219 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 14:58:19,219 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 14:58:19,219 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:58:19,219 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:58:19,219 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22319.90 MB 2025-02-14 14:58:19,219 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30754.75 MB 2025-02-14 14:58:19,219 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8434.85 MB 2025-02-14 14:58:19,219 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35991.32 MB 2025-02-14 14:58:19,219 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40183.53 MB 2025-02-14 14:58:19,219 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4192.21 MB 2025-02-14 14:58:19,219 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30754.75 MB 2025-02-14 14:58:19,381 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7950] 2025-02-14 14:58:19,383 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:58:19,383 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 14:58:19,383 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:58:19,384 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 14:58:19,388 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 14:58:19,389 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:58:19,389 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 14:58:19,389 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 14:58:28,366 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:58:28,366 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 14:58:28,373 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 14:58:28,378 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:58:28,378 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1894, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 14:58:28,380 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:58:28,380 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1894, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 14:58:58,087 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 14:58:58,087 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 14:58:58,087 - resource_logging.py:150 - __exit__ - DEBUG - Time: 29.70 seconds 2025-02-14 14:58:58,087 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:58:58,087 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26166.40 MB 2025-02-14 14:58:58,087 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32869.16 MB 2025-02-14 14:58:58,087 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6702.76 MB 2025-02-14 14:58:58,087 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52762.25 MB 2025-02-14 14:58:58,087 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40386.95 MB 2025-02-14 14:58:58,087 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12375.29 MB 2025-02-14 14:58:58,087 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41753.87 MB 2025-02-14 14:58:58,205 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 14:58:58,205 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 14:58:58,205 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 14:58:58,205 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:58:58,205 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32869.16 MB 2025-02-14 14:58:58,205 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25624.17 MB 2025-02-14 14:58:58,205 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7245.00 MB 2025-02-14 14:58:58,205 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40386.95 MB 2025-02-14 14:58:58,205 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 60416.85 MB 2025-02-14 14:58:58,205 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 20029.90 MB 2025-02-14 14:58:58,205 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51260.13 MB 2025-02-14 14:59:00,155 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 14:59:00,155 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 14:59:00,155 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.95 seconds 2025-02-14 14:59:00,155 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:59:00,155 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25624.17 MB 2025-02-14 14:59:00,155 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26155.01 MB 2025-02-14 14:59:00,155 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 14:59:00,155 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60416.85 MB 2025-02-14 14:59:00,155 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35097.94 MB 2025-02-14 14:59:00,155 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25318.92 MB 2025-02-14 14:59:00,155 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30134.34 MB 2025-02-14 14:59:00,168 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 14:59:00,168 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 14:59:00,168 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 14:59:00,168 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:59:00,168 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26155.01 MB 2025-02-14 14:59:00,168 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28044.54 MB 2025-02-14 14:59:00,168 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 14:59:00,168 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35097.94 MB 2025-02-14 14:59:00,168 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35097.94 MB 2025-02-14 14:59:00,168 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:59:00,168 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29461.97 MB 2025-02-14 14:59:00,386 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 14:59:00,386 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 14:59:00,386 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 14:59:00,386 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:59:00,386 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28044.54 MB 2025-02-14 14:59:00,386 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30287.42 MB 2025-02-14 14:59:00,386 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2242.88 MB 2025-02-14 14:59:00,386 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35097.94 MB 2025-02-14 14:59:00,386 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38400.95 MB 2025-02-14 14:59:00,386 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 14:59:00,386 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35831.70 MB 2025-02-14 14:59:00,386 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 14:59:00,386 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 14:59:00,386 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 14:59:00,386 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:59:00,386 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26155.01 MB 2025-02-14 14:59:00,386 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30287.42 MB 2025-02-14 14:59:00,386 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4132.41 MB 2025-02-14 14:59:00,386 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35097.94 MB 2025-02-14 14:59:00,386 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38400.95 MB 2025-02-14 14:59:00,387 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 14:59:00,387 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35831.70 MB 2025-02-14 14:59:00,597 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 14:59:00,597 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 14:59:00,597 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 14:59:00,597 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:59:00,597 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31820.96 MB 2025-02-14 14:59:00,597 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32587.96 MB 2025-02-14 14:59:00,597 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 14:59:00,597 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38400.95 MB 2025-02-14 14:59:00,597 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38814.09 MB 2025-02-14 14:59:00,597 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 14:59:00,597 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33295.75 MB 2025-02-14 14:59:00,618 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 14:59:00,618 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 14:59:00,618 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:59:00,618 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:59:00,618 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33000.85 MB 2025-02-14 14:59:00,618 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33229.52 MB 2025-02-14 14:59:00,618 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.67 MB 2025-02-14 14:59:00,618 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38814.09 MB 2025-02-14 14:59:00,618 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38814.09 MB 2025-02-14 14:59:00,618 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:59:00,618 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33435.33 MB 2025-02-14 14:59:00,619 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 14:59:00,619 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 14:59:00,619 - resource_logging.py:150 - __exit__ - DEBUG - Time: 32.24 seconds 2025-02-14 14:59:00,619 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:59:00,619 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19567.55 MB 2025-02-14 14:59:00,619 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33430.10 MB 2025-02-14 14:59:00,619 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13862.54 MB 2025-02-14 14:59:00,619 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52762.25 MB 2025-02-14 14:59:00,619 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38814.09 MB 2025-02-14 14:59:00,619 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13948.16 MB 2025-02-14 14:59:00,619 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33435.33 MB 2025-02-14 14:59:00,894 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 14:59:00,894 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 14:59:00,894 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 14:59:00,894 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:59:00,894 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33430.10 MB 2025-02-14 14:59:00,894 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24564.32 MB 2025-02-14 14:59:00,894 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8865.77 MB 2025-02-14 14:59:00,894 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38814.09 MB 2025-02-14 14:59:00,894 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38814.09 MB 2025-02-14 14:59:00,894 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 14:59:00,894 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35935.62 MB 2025-02-14 14:59:00,912 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8142, cut from 8144 2025-02-14 14:59:00,912 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 14:59:00,918 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 14:59:00,918 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 14:59:00,918 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 14:59:00,919 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 14:59:00,919 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24564.32 MB 2025-02-14 14:59:00,919 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32982.48 MB 2025-02-14 14:59:00,919 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8418.15 MB 2025-02-14 14:59:00,919 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38814.09 MB 2025-02-14 14:59:00,919 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42997.91 MB 2025-02-14 14:59:00,919 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4183.82 MB 2025-02-14 14:59:00,919 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32982.48 MB 2025-02-14 14:59:01,088 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7934] 2025-02-14 14:59:01,089 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:59:01,089 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 14:59:01,090 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:59:01,090 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 14:59:01,095 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 14:59:01,096 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 14:59:01,096 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 14:59:01,096 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 15:00:19,246 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:00:19,246 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:00:19,251 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 15:00:19,256 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:00:19,256 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 159, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 15:00:19,257 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:00:19,257 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 159, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 15:00:21,731 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 15:00:21,731 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 15:00:21,731 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.47 seconds 2025-02-14 15:00:21,731 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:00:21,731 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14076.64 MB 2025-02-14 15:00:21,731 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14639.34 MB 2025-02-14 15:00:21,731 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 562.69 MB 2025-02-14 15:00:21,731 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55551.46 MB 2025-02-14 15:00:21,731 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 16909.34 MB 2025-02-14 15:00:21,731 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -38642.12 MB 2025-02-14 15:00:21,731 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23548.82 MB 2025-02-14 15:00:21,744 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 15:00:21,744 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 15:00:21,744 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:00:21,744 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:00:21,744 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14639.34 MB 2025-02-14 15:00:21,744 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14911.96 MB 2025-02-14 15:00:21,744 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 272.62 MB 2025-02-14 15:00:21,744 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 16909.34 MB 2025-02-14 15:00:21,744 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18037.60 MB 2025-02-14 15:00:21,744 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1128.27 MB 2025-02-14 15:00:21,744 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16925.81 MB 2025-02-14 15:00:22,516 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 15:00:22,516 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 15:00:22,516 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.77 seconds 2025-02-14 15:00:22,516 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:00:22,516 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14911.96 MB 2025-02-14 15:00:22,516 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15122.97 MB 2025-02-14 15:00:22,516 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 211.01 MB 2025-02-14 15:00:22,516 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18037.60 MB 2025-02-14 15:00:22,516 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17473.47 MB 2025-02-14 15:00:22,516 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -564.13 MB 2025-02-14 15:00:22,516 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19083.43 MB 2025-02-14 15:00:22,523 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 15:00:22,523 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 15:00:22,523 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 15:00:22,523 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:00:22,523 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15122.90 MB 2025-02-14 15:00:22,523 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15873.81 MB 2025-02-14 15:00:22,523 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 750.91 MB 2025-02-14 15:00:22,523 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17473.47 MB 2025-02-14 15:00:22,523 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17473.47 MB 2025-02-14 15:00:22,523 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:00:22,523 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16437.82 MB 2025-02-14 15:00:22,611 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 15:00:22,611 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 15:00:22,611 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 15:00:22,611 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:00:22,611 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15873.81 MB 2025-02-14 15:00:22,611 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16765.25 MB 2025-02-14 15:00:22,611 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 891.44 MB 2025-02-14 15:00:22,611 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17473.47 MB 2025-02-14 15:00:22,611 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19725.81 MB 2025-02-14 15:00:22,611 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2252.34 MB 2025-02-14 15:00:22,611 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18970.69 MB 2025-02-14 15:00:22,612 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 15:00:22,612 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 15:00:22,612 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 15:00:22,612 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:00:22,612 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15122.90 MB 2025-02-14 15:00:22,612 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16765.25 MB 2025-02-14 15:00:22,612 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1642.35 MB 2025-02-14 15:00:22,612 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17473.47 MB 2025-02-14 15:00:22,612 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19725.81 MB 2025-02-14 15:00:22,612 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2252.34 MB 2025-02-14 15:00:22,612 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18970.69 MB 2025-02-14 15:00:22,680 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 15:00:22,680 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 15:00:22,680 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 15:00:22,680 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:00:22,680 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17374.83 MB 2025-02-14 15:00:22,680 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17680.29 MB 2025-02-14 15:00:22,680 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 305.46 MB 2025-02-14 15:00:22,680 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19725.81 MB 2025-02-14 15:00:22,680 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19887.29 MB 2025-02-14 15:00:22,680 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 161.48 MB 2025-02-14 15:00:22,680 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17968.12 MB 2025-02-14 15:00:22,690 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 15:00:22,690 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 15:00:22,690 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:00:22,690 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:00:22,690 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17844.42 MB 2025-02-14 15:00:22,690 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18064.30 MB 2025-02-14 15:00:22,690 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 219.87 MB 2025-02-14 15:00:22,690 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19887.29 MB 2025-02-14 15:00:22,690 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19887.29 MB 2025-02-14 15:00:22,690 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:00:22,690 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18078.99 MB 2025-02-14 15:00:22,691 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 15:00:22,691 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 15:00:22,691 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.43 seconds 2025-02-14 15:00:22,691 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:00:22,691 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13522.68 MB 2025-02-14 15:00:22,691 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18265.00 MB 2025-02-14 15:00:22,691 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4742.33 MB 2025-02-14 15:00:22,691 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55551.46 MB 2025-02-14 15:00:22,691 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19887.29 MB 2025-02-14 15:00:22,691 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35664.17 MB 2025-02-14 15:00:22,691 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18265.00 MB 2025-02-14 15:00:22,958 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 15:00:22,958 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 15:00:22,958 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 15:00:22,958 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:00:22,958 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18265.00 MB 2025-02-14 15:00:22,958 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17384.58 MB 2025-02-14 15:00:22,958 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -880.43 MB 2025-02-14 15:00:22,958 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19887.29 MB 2025-02-14 15:00:22,958 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20021.51 MB 2025-02-14 15:00:22,958 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 134.22 MB 2025-02-14 15:00:22,958 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19067.77 MB 2025-02-14 15:00:22,976 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8147, cut from 8149 2025-02-14 15:00:22,976 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2.'] 2025-02-14 15:00:22,982 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 15:00:22,982 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 15:00:22,982 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:00:22,982 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:00:22,982 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17384.58 MB 2025-02-14 15:00:22,982 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25807.78 MB 2025-02-14 15:00:22,982 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8423.21 MB 2025-02-14 15:00:22,982 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20021.51 MB 2025-02-14 15:00:22,982 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30492.59 MB 2025-02-14 15:00:22,982 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10471.08 MB 2025-02-14 15:00:22,982 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25807.78 MB 2025-02-14 15:00:23,144 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7939] 2025-02-14 15:00:23,145 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:00:23,145 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:00:23,146 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:00:23,146 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 15:00:23,151 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 15:00:23,152 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:00:23,152 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 15:00:23,152 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2.'] 2025-02-14 15:01:20,231 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:01:20,231 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:01:20,239 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 15:01:20,246 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:01:20,246 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1727, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 15:01:20,248 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:01:20,248 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1727, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 15:01:46,914 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 15:01:46,914 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 15:01:46,914 - resource_logging.py:150 - __exit__ - DEBUG - Time: 26.66 seconds 2025-02-14 15:01:46,914 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:01:46,914 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25002.72 MB 2025-02-14 15:01:46,914 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31114.41 MB 2025-02-14 15:01:46,914 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6111.69 MB 2025-02-14 15:01:46,914 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38868.62 MB 2025-02-14 15:01:46,914 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39772.49 MB 2025-02-14 15:01:46,914 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 903.87 MB 2025-02-14 15:01:46,914 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39992.57 MB 2025-02-14 15:01:47,018 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 15:01:47,018 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 15:01:47,018 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 15:01:47,018 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:01:47,018 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31114.41 MB 2025-02-14 15:01:47,018 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24755.99 MB 2025-02-14 15:01:47,018 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6358.42 MB 2025-02-14 15:01:47,018 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39772.49 MB 2025-02-14 15:01:47,018 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57310.97 MB 2025-02-14 15:01:47,018 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 17538.48 MB 2025-02-14 15:01:47,018 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48523.54 MB 2025-02-14 15:01:48,944 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 15:01:48,944 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 15:01:48,944 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 15:01:48,944 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:01:48,944 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24755.99 MB 2025-02-14 15:01:48,944 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25286.83 MB 2025-02-14 15:01:48,944 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 15:01:48,944 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57310.97 MB 2025-02-14 15:01:48,944 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30886.85 MB 2025-02-14 15:01:48,944 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26424.12 MB 2025-02-14 15:01:48,944 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29266.16 MB 2025-02-14 15:01:48,961 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 15:01:48,961 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 15:01:48,961 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:01:48,961 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:01:48,961 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25286.83 MB 2025-02-14 15:01:48,961 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27176.36 MB 2025-02-14 15:01:48,961 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 15:01:48,961 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30886.85 MB 2025-02-14 15:01:48,961 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31830.57 MB 2025-02-14 15:01:48,961 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 15:01:48,961 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28593.79 MB 2025-02-14 15:01:49,176 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 15:01:49,176 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 15:01:49,176 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 15:01:49,176 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:01:49,177 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27176.36 MB 2025-02-14 15:01:49,177 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29418.22 MB 2025-02-14 15:01:49,177 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 15:01:49,177 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31830.57 MB 2025-02-14 15:01:49,177 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37964.74 MB 2025-02-14 15:01:49,177 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 15:01:49,177 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34962.50 MB 2025-02-14 15:01:49,177 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 15:01:49,177 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 15:01:49,177 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 15:01:49,177 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:01:49,177 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25286.83 MB 2025-02-14 15:01:49,177 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29418.22 MB 2025-02-14 15:01:49,177 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 15:01:49,177 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30886.85 MB 2025-02-14 15:01:49,177 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37964.74 MB 2025-02-14 15:01:49,177 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7077.89 MB 2025-02-14 15:01:49,177 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34962.50 MB 2025-02-14 15:01:49,356 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 15:01:49,356 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 15:01:49,356 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 15:01:49,356 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:01:49,356 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30951.76 MB 2025-02-14 15:01:49,356 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31718.76 MB 2025-02-14 15:01:49,356 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 15:01:49,356 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37964.74 MB 2025-02-14 15:01:49,356 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38382.08 MB 2025-02-14 15:01:49,356 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 15:01:49,356 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32426.55 MB 2025-02-14 15:01:49,375 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 15:01:49,376 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 15:01:49,376 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:01:49,376 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:01:49,376 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32131.65 MB 2025-02-14 15:01:49,376 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32360.71 MB 2025-02-14 15:01:49,376 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.06 MB 2025-02-14 15:01:49,376 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38382.08 MB 2025-02-14 15:01:49,376 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38382.08 MB 2025-02-14 15:01:49,376 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:01:49,376 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32603.23 MB 2025-02-14 15:01:49,378 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 15:01:49,378 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 15:01:49,378 - resource_logging.py:150 - __exit__ - DEBUG - Time: 29.13 seconds 2025-02-14 15:01:49,378 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:01:49,378 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18985.71 MB 2025-02-14 15:01:49,378 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32561.68 MB 2025-02-14 15:01:49,378 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13575.97 MB 2025-02-14 15:01:49,378 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38868.62 MB 2025-02-14 15:01:49,378 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38382.08 MB 2025-02-14 15:01:49,378 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -486.54 MB 2025-02-14 15:01:49,378 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32603.23 MB 2025-02-14 15:01:49,647 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 15:01:49,647 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 15:01:49,647 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 15:01:49,647 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:01:49,647 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20976.02 MB 2025-02-14 15:01:49,647 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23988.58 MB 2025-02-14 15:01:49,647 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3012.56 MB 2025-02-14 15:01:49,647 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38382.08 MB 2025-02-14 15:01:49,647 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38382.08 MB 2025-02-14 15:01:49,647 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:01:49,647 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24289.80 MB 2025-02-14 15:01:49,665 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8158, cut from 8160 2025-02-14 15:01:49,665 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 1.'] 2025-02-14 15:01:49,671 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 15:01:49,671 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 15:01:49,671 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:01:49,671 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:01:49,671 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23988.58 MB 2025-02-14 15:01:49,671 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32423.43 MB 2025-02-14 15:01:49,671 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8434.85 MB 2025-02-14 15:01:49,672 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38382.08 MB 2025-02-14 15:01:49,672 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46768.59 MB 2025-02-14 15:01:49,672 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8386.51 MB 2025-02-14 15:01:49,672 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32423.43 MB 2025-02-14 15:01:49,836 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7950] 2025-02-14 15:01:49,838 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:01:49,838 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:01:49,839 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:01:49,839 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 15:01:49,844 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 15:01:49,845 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:01:49,845 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 15:01:49,845 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 1.'] 2025-02-14 15:04:04,855 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:04:04,855 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:04:04,863 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 15:04:04,871 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:04:04,871 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1440, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 15:04:04,873 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:04:04,873 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1440, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 15:04:26,982 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 15:04:26,982 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 15:04:26,982 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.10 seconds 2025-02-14 15:04:26,982 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:04:26,982 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23002.86 MB 2025-02-14 15:04:26,982 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28098.94 MB 2025-02-14 15:04:26,982 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5096.08 MB 2025-02-14 15:04:26,982 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59347.30 MB 2025-02-14 15:04:26,982 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38761.66 MB 2025-02-14 15:04:26,982 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20585.64 MB 2025-02-14 15:04:26,982 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37004.08 MB 2025-02-14 15:04:27,063 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 15:04:27,063 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 15:04:27,063 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 15:04:27,063 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:04:27,063 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28098.94 MB 2025-02-14 15:04:27,063 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23263.96 MB 2025-02-14 15:04:27,063 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4834.97 MB 2025-02-14 15:04:27,063 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38761.66 MB 2025-02-14 15:04:27,063 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48414.85 MB 2025-02-14 15:04:27,063 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9653.19 MB 2025-02-14 15:04:27,064 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42669.15 MB 2025-02-14 15:04:28,984 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 15:04:28,984 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 15:04:28,984 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 15:04:28,984 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:04:28,984 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23263.96 MB 2025-02-14 15:04:28,984 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23794.80 MB 2025-02-14 15:04:28,984 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 15:04:28,984 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48414.85 MB 2025-02-14 15:04:28,984 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29477.57 MB 2025-02-14 15:04:28,984 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18937.28 MB 2025-02-14 15:04:28,984 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27774.14 MB 2025-02-14 15:04:29,000 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 15:04:29,001 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 15:04:29,001 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:04:29,001 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:04:29,001 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23794.80 MB 2025-02-14 15:04:29,001 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25684.34 MB 2025-02-14 15:04:29,001 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 15:04:29,001 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29477.57 MB 2025-02-14 15:04:29,001 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30421.29 MB 2025-02-14 15:04:29,001 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 15:04:29,001 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27101.77 MB 2025-02-14 15:04:29,206 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 15:04:29,206 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 15:04:29,206 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 15:04:29,206 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:04:29,206 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25684.34 MB 2025-02-14 15:04:29,206 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27926.19 MB 2025-02-14 15:04:29,206 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 15:04:29,206 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30421.29 MB 2025-02-14 15:04:29,206 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36083.60 MB 2025-02-14 15:04:29,206 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 15:04:29,206 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33470.47 MB 2025-02-14 15:04:29,207 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 15:04:29,207 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 15:04:29,207 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 15:04:29,207 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:04:29,207 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23794.80 MB 2025-02-14 15:04:29,207 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27926.19 MB 2025-02-14 15:04:29,207 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 15:04:29,207 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29477.57 MB 2025-02-14 15:04:29,207 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36083.60 MB 2025-02-14 15:04:29,207 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 15:04:29,207 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33470.47 MB 2025-02-14 15:04:29,377 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 15:04:29,377 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 15:04:29,377 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 15:04:29,377 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:04:29,377 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29459.74 MB 2025-02-14 15:04:29,377 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30226.74 MB 2025-02-14 15:04:29,377 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 15:04:29,377 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36083.60 MB 2025-02-14 15:04:29,377 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36498.83 MB 2025-02-14 15:04:29,377 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 15:04:29,377 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30934.53 MB 2025-02-14 15:04:29,396 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 15:04:29,396 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 15:04:29,396 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:04:29,396 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:04:29,396 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30639.63 MB 2025-02-14 15:04:29,396 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30867.58 MB 2025-02-14 15:04:29,396 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.95 MB 2025-02-14 15:04:29,396 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36498.83 MB 2025-02-14 15:04:29,396 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36498.83 MB 2025-02-14 15:04:29,396 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:04:29,396 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31081.99 MB 2025-02-14 15:04:29,397 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 15:04:29,397 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 15:04:29,397 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.52 seconds 2025-02-14 15:04:29,397 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:04:29,397 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17985.78 MB 2025-02-14 15:04:29,397 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31067.45 MB 2025-02-14 15:04:29,397 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13081.66 MB 2025-02-14 15:04:29,397 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59347.30 MB 2025-02-14 15:04:29,397 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36498.83 MB 2025-02-14 15:04:29,397 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22848.47 MB 2025-02-14 15:04:29,397 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31081.99 MB 2025-02-14 15:04:29,663 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 15:04:29,663 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 15:04:29,663 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 15:04:29,663 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:04:29,663 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31067.45 MB 2025-02-14 15:04:29,663 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22971.50 MB 2025-02-14 15:04:29,663 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8095.94 MB 2025-02-14 15:04:29,663 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36498.83 MB 2025-02-14 15:04:29,663 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36498.83 MB 2025-02-14 15:04:29,663 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:04:29,663 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33564.06 MB 2025-02-14 15:04:29,680 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8113, cut from 8115 2025-02-14 15:04:29,681 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 15:04:29,687 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 15:04:29,687 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 15:04:29,687 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:04:29,687 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:04:29,687 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22971.50 MB 2025-02-14 15:04:29,687 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31359.92 MB 2025-02-14 15:04:29,687 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8388.42 MB 2025-02-14 15:04:29,687 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36498.83 MB 2025-02-14 15:04:29,687 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44839.21 MB 2025-02-14 15:04:29,687 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8340.37 MB 2025-02-14 15:04:29,687 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31359.92 MB 2025-02-14 15:04:29,837 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7905] 2025-02-14 15:04:29,839 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:04:29,839 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:04:29,840 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:04:29,840 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 15:04:29,844 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 15:04:29,845 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:04:29,845 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 15:04:29,846 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 15:05:52,805 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:05:52,805 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:05:52,810 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 15:05:52,814 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:05:52,814 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2934, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 15:05:52,815 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:05:52,815 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2934, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 15:06:38,195 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 15:06:38,195 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 15:06:38,195 - resource_logging.py:150 - __exit__ - DEBUG - Time: 45.37 seconds 2025-02-14 15:06:38,195 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:06:38,195 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33415.94 MB 2025-02-14 15:06:38,195 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43799.20 MB 2025-02-14 15:06:38,195 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 10383.26 MB 2025-02-14 15:06:38,195 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 77795.95 MB 2025-02-14 15:06:38,195 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47743.76 MB 2025-02-14 15:06:38,195 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30052.19 MB 2025-02-14 15:06:38,195 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 54182.46 MB 2025-02-14 15:06:38,394 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 15:06:38,394 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 15:06:38,394 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 15:06:38,394 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:06:38,394 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43799.20 MB 2025-02-14 15:06:38,394 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31032.53 MB 2025-02-14 15:06:38,394 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -12766.67 MB 2025-02-14 15:06:38,394 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47743.76 MB 2025-02-14 15:06:38,394 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 86748.69 MB 2025-02-14 15:06:38,394 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 39004.93 MB 2025-02-14 15:06:38,394 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 73534.59 MB 2025-02-14 15:06:40,378 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 15:06:40,378 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 15:06:40,378 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.98 seconds 2025-02-14 15:06:40,378 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:06:40,378 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31032.53 MB 2025-02-14 15:06:40,379 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31563.38 MB 2025-02-14 15:06:40,379 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 15:06:40,379 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 86748.69 MB 2025-02-14 15:06:40,379 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33581.69 MB 2025-02-14 15:06:40,379 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -53167.00 MB 2025-02-14 15:06:40,379 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35543.75 MB 2025-02-14 15:06:40,396 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 15:06:40,396 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 15:06:40,396 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:06:40,396 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:06:40,396 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31563.38 MB 2025-02-14 15:06:40,396 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33452.91 MB 2025-02-14 15:06:40,396 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 15:06:40,396 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33581.69 MB 2025-02-14 15:06:40,396 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36884.71 MB 2025-02-14 15:06:40,396 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 15:06:40,396 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34870.34 MB 2025-02-14 15:06:40,609 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 15:06:40,609 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 15:06:40,609 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 15:06:40,609 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:06:40,609 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33452.91 MB 2025-02-14 15:06:40,610 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35694.77 MB 2025-02-14 15:06:40,610 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 15:06:40,610 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36884.71 MB 2025-02-14 15:06:40,610 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43490.74 MB 2025-02-14 15:06:40,610 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 15:06:40,610 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41239.05 MB 2025-02-14 15:06:40,610 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 15:06:40,610 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 15:06:40,610 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 15:06:40,610 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:06:40,610 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31563.38 MB 2025-02-14 15:06:40,610 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35694.77 MB 2025-02-14 15:06:40,610 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 15:06:40,610 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33581.69 MB 2025-02-14 15:06:40,610 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43490.74 MB 2025-02-14 15:06:40,610 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9909.04 MB 2025-02-14 15:06:40,610 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41239.05 MB 2025-02-14 15:06:40,784 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 15:06:40,784 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 15:06:40,784 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 15:06:40,784 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:06:40,784 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37228.31 MB 2025-02-14 15:06:40,784 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37995.31 MB 2025-02-14 15:06:40,784 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 15:06:40,784 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43490.74 MB 2025-02-14 15:06:40,784 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43903.88 MB 2025-02-14 15:06:40,784 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 15:06:40,784 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38703.10 MB 2025-02-14 15:06:40,803 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 15:06:40,803 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 15:06:40,803 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:06:40,803 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:06:40,803 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38408.20 MB 2025-02-14 15:06:40,803 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38636.69 MB 2025-02-14 15:06:40,803 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.49 MB 2025-02-14 15:06:40,803 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43903.88 MB 2025-02-14 15:06:40,803 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43903.88 MB 2025-02-14 15:06:40,803 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:06:40,803 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38860.02 MB 2025-02-14 15:06:40,804 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 15:06:40,804 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 15:06:40,804 - resource_logging.py:150 - __exit__ - DEBUG - Time: 47.99 seconds 2025-02-14 15:06:40,804 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:06:40,804 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23192.32 MB 2025-02-14 15:06:40,804 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38836.98 MB 2025-02-14 15:06:40,804 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 15644.66 MB 2025-02-14 15:06:40,804 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 67572.33 MB 2025-02-14 15:06:40,804 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43903.88 MB 2025-02-14 15:06:40,804 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23668.46 MB 2025-02-14 15:06:40,804 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38860.02 MB 2025-02-14 15:06:41,074 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 15:06:41,075 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 15:06:41,075 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 15:06:41,075 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:06:41,075 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38836.98 MB 2025-02-14 15:06:41,075 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28184.52 MB 2025-02-14 15:06:41,075 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10652.46 MB 2025-02-14 15:06:41,075 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43903.88 MB 2025-02-14 15:06:41,075 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43903.88 MB 2025-02-14 15:06:41,075 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:06:41,075 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41338.82 MB 2025-02-14 15:06:41,092 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8130, cut from 8132 2025-02-14 15:06:41,093 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 15:06:41,099 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 15:06:41,099 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 15:06:41,099 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:06:41,099 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:06:41,099 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28184.52 MB 2025-02-14 15:06:41,099 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36590.18 MB 2025-02-14 15:06:41,099 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8405.66 MB 2025-02-14 15:06:41,099 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43903.88 MB 2025-02-14 15:06:41,099 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48083.50 MB 2025-02-14 15:06:41,099 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4179.62 MB 2025-02-14 15:06:41,099 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36590.18 MB 2025-02-14 15:06:41,261 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7922] 2025-02-14 15:06:41,262 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:06:41,262 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:06:41,263 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:06:41,263 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 15:06:41,268 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 15:06:41,269 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:06:41,269 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 15:06:41,269 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 15:06:53,081 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:06:53,081 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:06:53,086 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 15:06:53,090 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:06:53,090 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1825, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 15:06:53,091 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:06:53,091 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1825, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 15:07:21,656 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 15:07:21,657 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 15:07:21,657 - resource_logging.py:150 - __exit__ - DEBUG - Time: 28.56 seconds 2025-02-14 15:07:21,657 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:07:21,657 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25685.60 MB 2025-02-14 15:07:21,657 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32144.83 MB 2025-02-14 15:07:21,657 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6459.23 MB 2025-02-14 15:07:21,657 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56442.75 MB 2025-02-14 15:07:21,657 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40942.70 MB 2025-02-14 15:07:21,657 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15500.05 MB 2025-02-14 15:07:21,657 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41045.77 MB 2025-02-14 15:07:21,775 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 15:07:21,775 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 15:07:21,775 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 15:07:21,775 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:07:21,775 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32144.83 MB 2025-02-14 15:07:21,775 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25265.46 MB 2025-02-14 15:07:21,775 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6879.37 MB 2025-02-14 15:07:21,775 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40942.70 MB 2025-02-14 15:07:21,775 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 59617.84 MB 2025-02-14 15:07:21,775 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 18675.14 MB 2025-02-14 15:07:21,775 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51425.00 MB 2025-02-14 15:07:23,722 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 15:07:23,722 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 15:07:23,722 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.95 seconds 2025-02-14 15:07:23,722 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:07:23,722 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25265.46 MB 2025-02-14 15:07:23,722 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25796.30 MB 2025-02-14 15:07:23,722 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 15:07:23,722 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59617.84 MB 2025-02-14 15:07:23,722 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31719.42 MB 2025-02-14 15:07:23,722 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27898.41 MB 2025-02-14 15:07:23,722 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29775.63 MB 2025-02-14 15:07:23,736 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 15:07:23,736 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 15:07:23,736 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:07:23,736 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:07:23,736 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25796.30 MB 2025-02-14 15:07:23,736 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27685.83 MB 2025-02-14 15:07:23,736 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 15:07:23,736 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31719.42 MB 2025-02-14 15:07:23,736 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31719.42 MB 2025-02-14 15:07:23,736 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:07:23,736 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29103.26 MB 2025-02-14 15:07:23,954 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 15:07:23,954 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 15:07:23,954 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 15:07:23,954 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:07:23,954 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27685.83 MB 2025-02-14 15:07:23,954 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29927.69 MB 2025-02-14 15:07:23,954 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 15:07:23,954 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31719.42 MB 2025-02-14 15:07:23,954 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37853.59 MB 2025-02-14 15:07:23,954 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 15:07:23,954 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35471.97 MB 2025-02-14 15:07:23,955 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 15:07:23,955 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 15:07:23,955 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 15:07:23,955 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:07:23,955 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25796.30 MB 2025-02-14 15:07:23,955 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29927.69 MB 2025-02-14 15:07:23,955 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 15:07:23,955 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31719.42 MB 2025-02-14 15:07:23,955 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37853.59 MB 2025-02-14 15:07:23,955 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 15:07:23,955 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35471.97 MB 2025-02-14 15:07:24,132 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 15:07:24,132 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 15:07:24,132 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 15:07:24,132 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:07:24,132 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31461.23 MB 2025-02-14 15:07:24,132 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32228.23 MB 2025-02-14 15:07:24,132 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 15:07:24,132 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37853.59 MB 2025-02-14 15:07:24,132 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38264.64 MB 2025-02-14 15:07:24,132 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 411.04 MB 2025-02-14 15:07:24,132 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32936.02 MB 2025-02-14 15:07:24,151 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 15:07:24,152 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 15:07:24,152 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:07:24,152 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:07:24,152 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32641.12 MB 2025-02-14 15:07:24,152 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32869.28 MB 2025-02-14 15:07:24,152 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.16 MB 2025-02-14 15:07:24,152 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38264.64 MB 2025-02-14 15:07:24,152 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38264.64 MB 2025-02-14 15:07:24,152 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:07:24,152 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33115.67 MB 2025-02-14 15:07:24,153 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 15:07:24,153 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 15:07:24,153 - resource_logging.py:150 - __exit__ - DEBUG - Time: 31.06 seconds 2025-02-14 15:07:24,153 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:07:24,153 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19327.15 MB 2025-02-14 15:07:24,153 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33069.69 MB 2025-02-14 15:07:24,153 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13742.54 MB 2025-02-14 15:07:24,153 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56442.75 MB 2025-02-14 15:07:24,153 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38264.64 MB 2025-02-14 15:07:24,153 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18178.11 MB 2025-02-14 15:07:24,153 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33115.67 MB 2025-02-14 15:07:24,427 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 15:07:24,427 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 15:07:24,427 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 15:07:24,427 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:07:24,427 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33069.69 MB 2025-02-14 15:07:24,427 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24321.26 MB 2025-02-14 15:07:24,427 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8748.44 MB 2025-02-14 15:07:24,427 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38264.64 MB 2025-02-14 15:07:24,427 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38264.64 MB 2025-02-14 15:07:24,427 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:07:24,427 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35573.06 MB 2025-02-14 15:07:24,445 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8135, cut from 8137 2025-02-14 15:07:24,445 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-14 15:07:24,451 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 15:07:24,451 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 15:07:24,451 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:07:24,451 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:07:24,451 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24321.26 MB 2025-02-14 15:07:24,451 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32732.08 MB 2025-02-14 15:07:24,451 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8410.82 MB 2025-02-14 15:07:24,451 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38264.64 MB 2025-02-14 15:07:24,451 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42446.36 MB 2025-02-14 15:07:24,451 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4181.72 MB 2025-02-14 15:07:24,451 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32732.08 MB 2025-02-14 15:07:24,620 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7927] 2025-02-14 15:07:24,621 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:07:24,621 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:07:24,622 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:07:24,622 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 15:07:24,627 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 15:07:24,628 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:07:24,628 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 15:07:24,629 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-14 15:09:02,837 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:09:02,838 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:09:02,843 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 15:09:02,847 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:09:02,847 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 191, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 15:09:02,848 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:09:02,848 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 191, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 15:09:05,779 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 15:09:05,779 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 15:09:05,779 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.93 seconds 2025-02-14 15:09:05,779 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:09:05,779 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14299.63 MB 2025-02-14 15:09:05,779 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14975.56 MB 2025-02-14 15:09:05,779 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 675.94 MB 2025-02-14 15:09:05,779 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50809.80 MB 2025-02-14 15:09:05,779 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18324.91 MB 2025-02-14 15:09:05,779 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32484.88 MB 2025-02-14 15:09:05,779 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23853.72 MB 2025-02-14 15:09:05,794 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 15:09:05,794 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 15:09:05,794 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:09:05,794 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:09:05,794 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14975.56 MB 2025-02-14 15:09:05,794 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15205.39 MB 2025-02-14 15:09:05,794 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.82 MB 2025-02-14 15:09:05,794 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18324.91 MB 2025-02-14 15:09:05,794 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18951.96 MB 2025-02-14 15:09:05,794 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 627.05 MB 2025-02-14 15:09:05,794 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17515.52 MB 2025-02-14 15:09:06,634 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 15:09:06,634 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 15:09:06,634 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.84 seconds 2025-02-14 15:09:06,634 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:09:06,634 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15205.39 MB 2025-02-14 15:09:06,634 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15440.28 MB 2025-02-14 15:09:06,634 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 234.90 MB 2025-02-14 15:09:06,634 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18951.96 MB 2025-02-14 15:09:06,634 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19260.24 MB 2025-02-14 15:09:06,634 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 308.28 MB 2025-02-14 15:09:06,634 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19376.86 MB 2025-02-14 15:09:06,642 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 15:09:06,642 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 15:09:06,642 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:09:06,642 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:09:06,642 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15440.22 MB 2025-02-14 15:09:06,643 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16276.13 MB 2025-02-14 15:09:06,643 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 835.92 MB 2025-02-14 15:09:06,643 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19260.24 MB 2025-02-14 15:09:06,643 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19260.24 MB 2025-02-14 15:09:06,643 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:09:06,643 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16903.35 MB 2025-02-14 15:09:06,741 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 15:09:06,741 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 15:09:06,741 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 15:09:06,741 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:09:06,741 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16276.13 MB 2025-02-14 15:09:06,741 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17268.19 MB 2025-02-14 15:09:06,741 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 992.06 MB 2025-02-14 15:09:06,741 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19260.24 MB 2025-02-14 15:09:06,741 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20937.97 MB 2025-02-14 15:09:06,741 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1677.72 MB 2025-02-14 15:09:06,741 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19721.50 MB 2025-02-14 15:09:06,742 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 15:09:06,742 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 15:09:06,742 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 15:09:06,742 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:09:06,742 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15440.22 MB 2025-02-14 15:09:06,742 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17268.19 MB 2025-02-14 15:09:06,742 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1827.97 MB 2025-02-14 15:09:06,742 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19260.24 MB 2025-02-14 15:09:06,742 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20937.97 MB 2025-02-14 15:09:06,742 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1677.72 MB 2025-02-14 15:09:06,742 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19721.50 MB 2025-02-14 15:09:06,818 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 15:09:06,818 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 15:09:06,818 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 15:09:06,818 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:09:06,818 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17946.79 MB 2025-02-14 15:09:06,818 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18286.18 MB 2025-02-14 15:09:06,818 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 339.40 MB 2025-02-14 15:09:06,818 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20937.97 MB 2025-02-14 15:09:06,818 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21118.32 MB 2025-02-14 15:09:06,818 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 180.36 MB 2025-02-14 15:09:06,818 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18605.61 MB 2025-02-14 15:09:06,828 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 15:09:06,828 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 15:09:06,828 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:09:06,828 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:09:06,828 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18468.89 MB 2025-02-14 15:09:06,828 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18696.14 MB 2025-02-14 15:09:06,828 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.25 MB 2025-02-14 15:09:06,828 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21118.32 MB 2025-02-14 15:09:06,828 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21118.32 MB 2025-02-14 15:09:06,828 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:09:06,828 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18721.99 MB 2025-02-14 15:09:06,829 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 15:09:06,829 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 15:09:06,829 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.98 seconds 2025-02-14 15:09:06,829 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:09:06,829 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13634.17 MB 2025-02-14 15:09:06,829 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18897.21 MB 2025-02-14 15:09:06,829 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5263.05 MB 2025-02-14 15:09:06,829 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50809.80 MB 2025-02-14 15:09:06,829 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21118.32 MB 2025-02-14 15:09:06,829 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29691.48 MB 2025-02-14 15:09:06,829 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18897.21 MB 2025-02-14 15:09:07,093 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 15:09:07,094 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 15:09:07,094 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 15:09:07,094 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:09:07,094 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18897.21 MB 2025-02-14 15:09:07,094 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17585.89 MB 2025-02-14 15:09:07,094 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1311.32 MB 2025-02-14 15:09:07,094 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21118.32 MB 2025-02-14 15:09:07,094 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21118.32 MB 2025-02-14 15:09:07,094 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:09:07,094 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19131.63 MB 2025-02-14 15:09:07,111 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 15:09:07,112 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 15:09:07,118 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 15:09:07,118 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 15:09:07,118 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:09:07,118 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:09:07,118 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17585.89 MB 2025-02-14 15:09:07,118 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26024.91 MB 2025-02-14 15:09:07,118 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 15:09:07,118 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21118.32 MB 2025-02-14 15:09:07,118 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31608.27 MB 2025-02-14 15:09:07,118 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 15:09:07,118 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26024.91 MB 2025-02-14 15:09:07,280 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 15:09:07,281 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:09:07,281 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:09:07,282 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:09:07,282 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 15:09:07,287 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 15:09:07,288 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:09:07,288 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 15:09:07,288 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 15:09:59,264 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:09:59,264 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:09:59,269 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 15:09:59,273 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:09:59,273 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1762, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 15:09:59,274 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:09:59,274 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1762, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 15:10:26,360 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 15:10:26,360 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 15:10:26,360 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.08 seconds 2025-02-14 15:10:26,360 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:10:26,360 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25246.60 MB 2025-02-14 15:10:26,360 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31482.22 MB 2025-02-14 15:10:26,360 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6235.62 MB 2025-02-14 15:10:26,360 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44193.28 MB 2025-02-14 15:10:26,360 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39925.58 MB 2025-02-14 15:10:26,360 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4267.70 MB 2025-02-14 15:10:26,360 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40381.09 MB 2025-02-14 15:10:26,489 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 15:10:26,489 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 15:10:26,489 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 15:10:26,489 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:10:26,489 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31482.22 MB 2025-02-14 15:10:26,489 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24937.94 MB 2025-02-14 15:10:26,490 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6544.28 MB 2025-02-14 15:10:26,490 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39925.58 MB 2025-02-14 15:10:26,490 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57646.51 MB 2025-02-14 15:10:26,490 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 17720.93 MB 2025-02-14 15:10:26,490 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48845.69 MB 2025-02-14 15:10:28,416 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 15:10:28,416 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 15:10:28,416 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 15:10:28,416 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:10:28,416 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24937.94 MB 2025-02-14 15:10:28,416 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25468.78 MB 2025-02-14 15:10:28,416 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 15:10:28,416 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57646.51 MB 2025-02-14 15:10:28,416 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30909.92 MB 2025-02-14 15:10:28,416 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26736.59 MB 2025-02-14 15:10:28,416 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29448.11 MB 2025-02-14 15:10:28,433 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 15:10:28,433 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 15:10:28,433 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:10:28,433 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:10:28,433 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25468.78 MB 2025-02-14 15:10:28,433 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27358.31 MB 2025-02-14 15:10:28,433 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 15:10:28,433 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30909.92 MB 2025-02-14 15:10:28,433 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31853.64 MB 2025-02-14 15:10:28,433 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 15:10:28,433 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28775.74 MB 2025-02-14 15:10:28,645 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 15:10:28,645 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 15:10:28,645 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 15:10:28,645 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:10:28,645 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27358.31 MB 2025-02-14 15:10:28,645 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29600.17 MB 2025-02-14 15:10:28,645 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 15:10:28,645 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31853.64 MB 2025-02-14 15:10:28,645 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37987.81 MB 2025-02-14 15:10:28,645 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 15:10:28,645 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35144.45 MB 2025-02-14 15:10:28,646 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 15:10:28,646 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 15:10:28,646 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 15:10:28,646 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:10:28,646 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25468.78 MB 2025-02-14 15:10:28,646 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29600.17 MB 2025-02-14 15:10:28,646 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 15:10:28,646 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30909.92 MB 2025-02-14 15:10:28,646 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37987.81 MB 2025-02-14 15:10:28,646 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7077.89 MB 2025-02-14 15:10:28,646 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35144.45 MB 2025-02-14 15:10:28,815 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 15:10:28,815 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 15:10:28,815 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 15:10:28,815 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:10:28,815 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31133.71 MB 2025-02-14 15:10:28,815 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31900.71 MB 2025-02-14 15:10:28,815 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 15:10:28,815 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37987.81 MB 2025-02-14 15:10:28,815 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38403.05 MB 2025-02-14 15:10:28,815 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 15:10:28,815 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32608.50 MB 2025-02-14 15:10:28,834 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 15:10:28,834 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 15:10:28,834 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:10:28,834 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:10:28,834 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32313.60 MB 2025-02-14 15:10:28,834 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32541.89 MB 2025-02-14 15:10:28,834 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.28 MB 2025-02-14 15:10:28,834 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38403.05 MB 2025-02-14 15:10:28,834 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38403.05 MB 2025-02-14 15:10:28,834 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:10:28,834 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32764.24 MB 2025-02-14 15:10:28,836 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 15:10:28,836 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 15:10:28,836 - resource_logging.py:150 - __exit__ - DEBUG - Time: 29.56 seconds 2025-02-14 15:10:28,836 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:10:28,836 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19107.66 MB 2025-02-14 15:10:28,836 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32742.59 MB 2025-02-14 15:10:28,836 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13634.94 MB 2025-02-14 15:10:28,836 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44193.28 MB 2025-02-14 15:10:28,836 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38403.05 MB 2025-02-14 15:10:28,836 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5790.24 MB 2025-02-14 15:10:28,836 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32764.24 MB 2025-02-14 15:10:29,104 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 15:10:29,104 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 15:10:29,104 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 15:10:29,104 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:10:29,104 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32742.59 MB 2025-02-14 15:10:29,104 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24106.33 MB 2025-02-14 15:10:29,104 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8636.26 MB 2025-02-14 15:10:29,104 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38403.05 MB 2025-02-14 15:10:29,104 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38403.05 MB 2025-02-14 15:10:29,104 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:10:29,104 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35249.65 MB 2025-02-14 15:10:29,122 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8147, cut from 8149 2025-02-14 15:10:29,122 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 15:10:29,128 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 15:10:29,128 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 15:10:29,128 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:10:29,128 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:10:29,128 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24106.33 MB 2025-02-14 15:10:29,128 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32529.54 MB 2025-02-14 15:10:29,128 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8423.21 MB 2025-02-14 15:10:29,128 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38403.05 MB 2025-02-14 15:10:29,128 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46779.07 MB 2025-02-14 15:10:29,128 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8376.03 MB 2025-02-14 15:10:29,128 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32529.54 MB 2025-02-14 15:10:29,292 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7939] 2025-02-14 15:10:29,294 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:10:29,294 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:10:29,295 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:10:29,295 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 15:10:29,299 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 15:10:29,300 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:10:29,300 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 15:10:29,301 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 15:10:38,864 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:10:38,864 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:10:38,869 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 15:10:38,872 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:10:38,872 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1097, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 15:10:38,873 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:10:38,873 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1097, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 15:10:56,014 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 15:10:56,014 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 15:10:56,014 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.13 seconds 2025-02-14 15:10:56,014 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:10:56,014 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20612.78 MB 2025-02-14 15:10:56,014 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24495.00 MB 2025-02-14 15:10:56,014 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3882.22 MB 2025-02-14 15:10:56,014 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55155.10 MB 2025-02-14 15:10:56,014 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29177.68 MB 2025-02-14 15:10:56,014 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25977.42 MB 2025-02-14 15:10:56,014 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33482.34 MB 2025-02-14 15:10:56,084 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 15:10:56,084 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 15:10:56,084 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 15:10:56,084 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:10:56,084 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24495.00 MB 2025-02-14 15:10:56,084 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21480.81 MB 2025-02-14 15:10:56,084 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3014.19 MB 2025-02-14 15:10:56,084 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29177.68 MB 2025-02-14 15:10:56,084 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42110.81 MB 2025-02-14 15:10:56,084 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 12933.14 MB 2025-02-14 15:10:56,084 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35993.10 MB 2025-02-14 15:10:58,010 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 15:10:58,010 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 15:10:58,010 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 15:10:58,010 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:10:58,010 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21480.81 MB 2025-02-14 15:10:58,010 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22011.65 MB 2025-02-14 15:10:58,010 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 15:10:58,010 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42110.81 MB 2025-02-14 15:10:58,010 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26709.33 MB 2025-02-14 15:10:58,010 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15401.48 MB 2025-02-14 15:10:58,010 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25990.99 MB 2025-02-14 15:10:58,025 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 15:10:58,025 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 15:10:58,025 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:10:58,025 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:10:58,025 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22011.65 MB 2025-02-14 15:10:58,025 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23901.19 MB 2025-02-14 15:10:58,025 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 15:10:58,025 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26709.33 MB 2025-02-14 15:10:58,025 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28596.76 MB 2025-02-14 15:10:58,025 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 15:10:58,025 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25318.62 MB 2025-02-14 15:10:58,240 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 15:10:58,240 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 15:10:58,240 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 15:10:58,240 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:10:58,240 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23901.19 MB 2025-02-14 15:10:58,240 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26143.04 MB 2025-02-14 15:10:58,240 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 15:10:58,240 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28596.76 MB 2025-02-14 15:10:58,240 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34259.08 MB 2025-02-14 15:10:58,240 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 15:10:58,240 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31687.32 MB 2025-02-14 15:10:58,241 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 15:10:58,241 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 15:10:58,241 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 15:10:58,241 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:10:58,241 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22011.65 MB 2025-02-14 15:10:58,241 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26143.04 MB 2025-02-14 15:10:58,241 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 15:10:58,241 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26709.33 MB 2025-02-14 15:10:58,241 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34259.08 MB 2025-02-14 15:10:58,241 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-14 15:10:58,241 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31687.32 MB 2025-02-14 15:10:58,408 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 15:10:58,408 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 15:10:58,408 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 15:10:58,408 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:10:58,408 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27676.59 MB 2025-02-14 15:10:58,408 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28443.59 MB 2025-02-14 15:10:58,408 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 15:10:58,408 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34259.08 MB 2025-02-14 15:10:58,408 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34674.31 MB 2025-02-14 15:10:58,408 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 15:10:58,408 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29151.38 MB 2025-02-14 15:10:58,427 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 15:10:58,427 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 15:10:58,427 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:10:58,427 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:10:58,427 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28856.48 MB 2025-02-14 15:10:58,427 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29084.60 MB 2025-02-14 15:10:58,427 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.13 MB 2025-02-14 15:10:58,427 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34674.31 MB 2025-02-14 15:10:58,427 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34674.31 MB 2025-02-14 15:10:58,427 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:10:58,427 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29295.59 MB 2025-02-14 15:10:58,428 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 15:10:58,428 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 15:10:58,428 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.55 seconds 2025-02-14 15:10:58,428 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:10:58,428 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16790.74 MB 2025-02-14 15:10:58,428 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29284.64 MB 2025-02-14 15:10:58,428 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12493.90 MB 2025-02-14 15:10:58,428 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55155.10 MB 2025-02-14 15:10:58,428 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34674.31 MB 2025-02-14 15:10:58,428 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20480.79 MB 2025-02-14 15:10:58,428 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29295.59 MB 2025-02-14 15:10:58,697 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 15:10:58,697 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 15:10:58,697 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 15:10:58,697 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:10:58,697 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29284.64 MB 2025-02-14 15:10:58,697 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21779.13 MB 2025-02-14 15:10:58,697 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7505.51 MB 2025-02-14 15:10:58,697 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34674.31 MB 2025-02-14 15:10:58,697 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34674.31 MB 2025-02-14 15:10:58,697 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:10:58,697 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31783.41 MB 2025-02-14 15:10:58,715 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8120, cut from 8122 2025-02-14 15:10:58,715 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 15:10:58,721 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 15:10:58,721 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 15:10:58,721 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:10:58,721 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:10:58,721 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21779.13 MB 2025-02-14 15:10:58,721 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30174.87 MB 2025-02-14 15:10:58,721 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8395.73 MB 2025-02-14 15:10:58,721 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34674.31 MB 2025-02-14 15:10:58,721 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38847.64 MB 2025-02-14 15:10:58,721 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4173.33 MB 2025-02-14 15:10:58,721 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30174.87 MB 2025-02-14 15:10:58,882 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7912] 2025-02-14 15:10:58,884 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:10:58,884 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:10:58,885 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:10:58,885 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 15:10:58,890 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 15:10:58,891 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:10:58,891 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 15:10:58,891 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 15:11:58,697 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:11:58,698 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:11:58,703 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 15:11:58,706 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:11:58,706 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 181, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 15:11:58,707 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:11:58,707 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 181, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 15:12:01,495 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 15:12:01,495 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 15:12:01,495 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.78 seconds 2025-02-14 15:12:01,495 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:12:01,495 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14229.94 MB 2025-02-14 15:12:01,495 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14870.49 MB 2025-02-14 15:12:01,495 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 640.55 MB 2025-02-14 15:12:01,495 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47194.31 MB 2025-02-14 15:12:01,495 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17853.05 MB 2025-02-14 15:12:01,495 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29341.25 MB 2025-02-14 15:12:01,495 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23702.12 MB 2025-02-14 15:12:01,509 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 15:12:01,509 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 15:12:01,509 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:12:01,509 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:12:01,509 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14870.49 MB 2025-02-14 15:12:01,509 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15090.59 MB 2025-02-14 15:12:01,509 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 220.09 MB 2025-02-14 15:12:01,509 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17853.05 MB 2025-02-14 15:12:01,509 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18448.65 MB 2025-02-14 15:12:01,509 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 595.59 MB 2025-02-14 15:12:01,509 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17249.04 MB 2025-02-14 15:12:02,316 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 15:12:02,316 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 15:12:02,316 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.81 seconds 2025-02-14 15:12:02,316 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:12:02,316 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15090.59 MB 2025-02-14 15:12:02,316 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15313.54 MB 2025-02-14 15:12:02,316 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 222.95 MB 2025-02-14 15:12:02,316 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18448.65 MB 2025-02-14 15:12:02,316 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 17976.79 MB 2025-02-14 15:12:02,316 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -471.86 MB 2025-02-14 15:12:02,316 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19262.06 MB 2025-02-14 15:12:02,324 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 15:12:02,324 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 15:12:02,324 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:12:02,324 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:12:02,324 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15313.47 MB 2025-02-14 15:12:02,324 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16106.89 MB 2025-02-14 15:12:02,324 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 793.41 MB 2025-02-14 15:12:02,324 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17976.79 MB 2025-02-14 15:12:02,324 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18373.15 MB 2025-02-14 15:12:02,324 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 396.36 MB 2025-02-14 15:12:02,324 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16702.21 MB 2025-02-14 15:12:02,415 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 15:12:02,415 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 15:12:02,415 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 15:12:02,415 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:12:02,415 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16106.89 MB 2025-02-14 15:12:02,415 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17048.50 MB 2025-02-14 15:12:02,415 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 941.62 MB 2025-02-14 15:12:02,415 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18373.15 MB 2025-02-14 15:12:02,415 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20554.19 MB 2025-02-14 15:12:02,415 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2181.04 MB 2025-02-14 15:12:02,415 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19378.11 MB 2025-02-14 15:12:02,416 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 15:12:02,416 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 15:12:02,416 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 15:12:02,416 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:12:02,416 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15313.47 MB 2025-02-14 15:12:02,416 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17048.50 MB 2025-02-14 15:12:02,416 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1735.03 MB 2025-02-14 15:12:02,416 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 17976.79 MB 2025-02-14 15:12:02,416 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20554.19 MB 2025-02-14 15:12:02,416 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2577.40 MB 2025-02-14 15:12:02,416 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19378.11 MB 2025-02-14 15:12:02,488 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 15:12:02,488 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 15:12:02,488 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 15:12:02,488 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:12:02,488 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17692.59 MB 2025-02-14 15:12:02,488 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18015.78 MB 2025-02-14 15:12:02,488 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 323.19 MB 2025-02-14 15:12:02,488 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20554.19 MB 2025-02-14 15:12:02,488 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20726.15 MB 2025-02-14 15:12:02,488 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 171.97 MB 2025-02-14 15:12:02,488 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18319.01 MB 2025-02-14 15:12:02,498 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 15:12:02,498 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 15:12:02,498 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:12:02,498 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:12:02,498 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18189.20 MB 2025-02-14 15:12:02,498 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18416.24 MB 2025-02-14 15:12:02,498 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.04 MB 2025-02-14 15:12:02,498 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20726.15 MB 2025-02-14 15:12:02,498 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20726.15 MB 2025-02-14 15:12:02,498 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:12:02,498 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18442.07 MB 2025-02-14 15:12:02,499 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 15:12:02,499 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 15:12:02,499 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.79 seconds 2025-02-14 15:12:02,499 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:12:02,499 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13599.32 MB 2025-02-14 15:12:02,499 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18616.97 MB 2025-02-14 15:12:02,499 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5017.64 MB 2025-02-14 15:12:02,499 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47194.31 MB 2025-02-14 15:12:02,499 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20726.15 MB 2025-02-14 15:12:02,499 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26468.16 MB 2025-02-14 15:12:02,499 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18616.97 MB 2025-02-14 15:12:02,765 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 15:12:02,765 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 15:12:02,765 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 15:12:02,765 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:12:02,765 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18616.97 MB 2025-02-14 15:12:02,765 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17504.55 MB 2025-02-14 15:12:02,765 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1112.41 MB 2025-02-14 15:12:02,765 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20726.15 MB 2025-02-14 15:12:02,765 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20726.15 MB 2025-02-14 15:12:02,765 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:12:02,765 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19218.74 MB 2025-02-14 15:12:02,783 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8148, cut from 8150 2025-02-14 15:12:02,783 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 15:12:02,789 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 15:12:02,790 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 15:12:02,790 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:12:02,790 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:12:02,790 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17504.55 MB 2025-02-14 15:12:02,790 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25929.50 MB 2025-02-14 15:12:02,790 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8424.95 MB 2025-02-14 15:12:02,790 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20726.15 MB 2025-02-14 15:12:02,790 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31197.23 MB 2025-02-14 15:12:02,790 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10471.08 MB 2025-02-14 15:12:02,790 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25929.50 MB 2025-02-14 15:12:02,952 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7940] 2025-02-14 15:12:02,954 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:12:02,954 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:12:02,955 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:12:02,955 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 15:12:02,960 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 15:12:02,961 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:12:02,961 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 15:12:02,961 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 15:13:09,736 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:13:09,736 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:13:09,741 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 15:13:09,745 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:13:09,745 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1265, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 15:13:09,746 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:13:09,746 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1265, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 15:13:29,135 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 15:13:29,136 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 15:13:29,136 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.38 seconds 2025-02-14 15:13:29,136 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:13:29,136 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21783.43 MB 2025-02-14 15:13:29,136 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26260.85 MB 2025-02-14 15:13:29,136 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4477.42 MB 2025-02-14 15:13:29,136 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39573.26 MB 2025-02-14 15:13:29,136 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38138.81 MB 2025-02-14 15:13:29,136 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1434.45 MB 2025-02-14 15:13:29,136 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35105.17 MB 2025-02-14 15:13:29,211 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 15:13:29,211 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 15:13:29,211 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 15:13:29,211 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:13:29,211 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26260.85 MB 2025-02-14 15:13:29,211 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22354.19 MB 2025-02-14 15:13:29,211 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3906.66 MB 2025-02-14 15:13:29,212 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38138.81 MB 2025-02-14 15:13:29,212 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46955.23 MB 2025-02-14 15:13:29,212 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8816.43 MB 2025-02-14 15:13:29,212 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39521.27 MB 2025-02-14 15:13:31,121 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 15:13:31,121 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 15:13:31,121 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 15:13:31,121 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:13:31,121 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22354.19 MB 2025-02-14 15:13:31,121 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22885.03 MB 2025-02-14 15:13:31,121 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 15:13:31,121 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46955.23 MB 2025-02-14 15:13:31,121 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33661.39 MB 2025-02-14 15:13:31,121 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13293.85 MB 2025-02-14 15:13:31,121 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26864.37 MB 2025-02-14 15:13:31,134 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 15:13:31,135 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 15:13:31,135 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:13:31,135 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:13:31,135 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22885.03 MB 2025-02-14 15:13:31,135 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24774.57 MB 2025-02-14 15:13:31,135 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 15:13:31,135 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33661.39 MB 2025-02-14 15:13:31,135 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33661.39 MB 2025-02-14 15:13:31,135 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:13:31,135 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26192.00 MB 2025-02-14 15:13:31,343 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 15:13:31,343 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 15:13:31,343 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 15:13:31,343 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:13:31,343 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24774.57 MB 2025-02-14 15:13:31,343 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27016.42 MB 2025-02-14 15:13:31,343 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 15:13:31,343 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33661.39 MB 2025-02-14 15:13:31,343 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35548.82 MB 2025-02-14 15:13:31,343 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 15:13:31,343 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32560.70 MB 2025-02-14 15:13:31,344 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 15:13:31,344 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 15:13:31,344 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 15:13:31,344 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:13:31,344 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22885.03 MB 2025-02-14 15:13:31,344 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27016.42 MB 2025-02-14 15:13:31,344 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 15:13:31,344 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33661.39 MB 2025-02-14 15:13:31,344 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35548.82 MB 2025-02-14 15:13:31,344 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 15:13:31,344 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32560.70 MB 2025-02-14 15:13:31,510 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 15:13:31,510 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 15:13:31,510 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 15:13:31,510 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:13:31,510 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28549.97 MB 2025-02-14 15:13:31,510 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29316.97 MB 2025-02-14 15:13:31,510 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 15:13:31,511 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35548.82 MB 2025-02-14 15:13:31,511 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35964.06 MB 2025-02-14 15:13:31,511 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 15:13:31,511 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30024.76 MB 2025-02-14 15:13:31,529 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 15:13:31,529 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 15:13:31,529 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:13:31,529 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:13:31,529 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29729.86 MB 2025-02-14 15:13:31,529 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29957.32 MB 2025-02-14 15:13:31,529 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.46 MB 2025-02-14 15:13:31,529 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35964.06 MB 2025-02-14 15:13:31,529 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35964.06 MB 2025-02-14 15:13:31,529 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:13:31,529 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30192.33 MB 2025-02-14 15:13:31,531 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 15:13:31,531 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 15:13:31,531 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.78 seconds 2025-02-14 15:13:31,531 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:13:31,531 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17376.07 MB 2025-02-14 15:13:31,531 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30157.28 MB 2025-02-14 15:13:31,531 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12781.22 MB 2025-02-14 15:13:31,531 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39573.26 MB 2025-02-14 15:13:31,531 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35964.06 MB 2025-02-14 15:13:31,531 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3609.20 MB 2025-02-14 15:13:31,531 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30192.33 MB 2025-02-14 15:13:31,798 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 15:13:31,798 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 15:13:31,798 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 15:13:31,798 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:13:31,798 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30157.28 MB 2025-02-14 15:13:31,798 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22363.32 MB 2025-02-14 15:13:31,798 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7793.97 MB 2025-02-14 15:13:31,798 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35964.06 MB 2025-02-14 15:13:31,798 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35964.06 MB 2025-02-14 15:13:31,798 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:13:31,798 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32655.13 MB 2025-02-14 15:13:31,816 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8117, cut from 8119 2025-02-14 15:13:31,816 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 15:13:31,822 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 15:13:31,822 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 15:13:31,822 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:13:31,822 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:13:31,822 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22363.32 MB 2025-02-14 15:13:31,822 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30755.91 MB 2025-02-14 15:13:31,822 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8392.59 MB 2025-02-14 15:13:31,822 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35964.06 MB 2025-02-14 15:13:31,822 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40135.29 MB 2025-02-14 15:13:31,822 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4171.24 MB 2025-02-14 15:13:31,822 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30755.91 MB 2025-02-14 15:13:31,976 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7909] 2025-02-14 15:13:31,977 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:13:31,977 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:13:31,978 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:13:31,978 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 15:13:31,982 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 15:13:31,983 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:13:31,983 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 15:13:31,984 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 15:14:25,649 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:14:25,649 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:14:25,654 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 15:14:25,659 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:14:25,659 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1650, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 15:14:25,660 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:14:25,660 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1650, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 15:14:51,094 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 15:14:51,094 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 15:14:51,094 - resource_logging.py:150 - __exit__ - DEBUG - Time: 25.42 seconds 2025-02-14 15:14:51,094 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:14:51,094 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24466.17 MB 2025-02-14 15:14:51,094 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30305.43 MB 2025-02-14 15:14:51,094 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5839.26 MB 2025-02-14 15:14:51,094 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52651.10 MB 2025-02-14 15:14:51,094 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39436.94 MB 2025-02-14 15:14:51,094 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13214.15 MB 2025-02-14 15:14:51,094 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39147.67 MB 2025-02-14 15:14:51,196 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 15:14:51,196 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 15:14:51,196 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 15:14:51,197 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:14:51,197 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30305.43 MB 2025-02-14 15:14:51,197 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24355.69 MB 2025-02-14 15:14:51,197 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5949.74 MB 2025-02-14 15:14:51,197 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39436.94 MB 2025-02-14 15:14:51,197 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56329.50 MB 2025-02-14 15:14:51,197 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 16892.56 MB 2025-02-14 15:14:51,197 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47509.76 MB 2025-02-14 15:14:53,130 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 15:14:53,130 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 15:14:53,130 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 15:14:53,130 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:14:53,131 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24355.69 MB 2025-02-14 15:14:53,131 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24886.53 MB 2025-02-14 15:14:53,131 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 15:14:53,131 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56329.50 MB 2025-02-14 15:14:53,131 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30840.72 MB 2025-02-14 15:14:53,131 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25488.79 MB 2025-02-14 15:14:53,131 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28865.86 MB 2025-02-14 15:14:53,147 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 15:14:53,147 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 15:14:53,147 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:14:53,147 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:14:53,147 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24886.53 MB 2025-02-14 15:14:53,147 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26776.06 MB 2025-02-14 15:14:53,147 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 15:14:53,147 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30840.72 MB 2025-02-14 15:14:53,147 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31784.44 MB 2025-02-14 15:14:53,147 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 15:14:53,147 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28193.49 MB 2025-02-14 15:14:53,361 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 15:14:53,361 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 15:14:53,361 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 15:14:53,361 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:14:53,361 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26776.06 MB 2025-02-14 15:14:53,361 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29017.92 MB 2025-02-14 15:14:53,361 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 15:14:53,361 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31784.44 MB 2025-02-14 15:14:53,361 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37446.75 MB 2025-02-14 15:14:53,361 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 15:14:53,361 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34562.20 MB 2025-02-14 15:14:53,362 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 15:14:53,362 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 15:14:53,362 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 15:14:53,362 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:14:53,362 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24886.53 MB 2025-02-14 15:14:53,362 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29017.92 MB 2025-02-14 15:14:53,362 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 15:14:53,362 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30840.72 MB 2025-02-14 15:14:53,362 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37446.75 MB 2025-02-14 15:14:53,362 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 15:14:53,362 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34562.20 MB 2025-02-14 15:14:53,537 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 15:14:53,537 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 15:14:53,537 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 15:14:53,537 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:14:53,537 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30551.46 MB 2025-02-14 15:14:53,538 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31318.46 MB 2025-02-14 15:14:53,538 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 15:14:53,538 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37446.75 MB 2025-02-14 15:14:53,538 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37861.98 MB 2025-02-14 15:14:53,538 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 15:14:53,538 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32026.25 MB 2025-02-14 15:14:53,558 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 15:14:53,558 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 15:14:53,558 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:14:53,558 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:14:53,558 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31731.35 MB 2025-02-14 15:14:53,558 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31959.38 MB 2025-02-14 15:14:53,558 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.03 MB 2025-02-14 15:14:53,558 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37861.98 MB 2025-02-14 15:14:53,558 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37861.98 MB 2025-02-14 15:14:53,558 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:14:53,558 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32201.73 MB 2025-02-14 15:14:53,559 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 15:14:53,559 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 15:14:53,560 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.90 seconds 2025-02-14 15:14:53,560 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:14:53,560 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18717.44 MB 2025-02-14 15:14:53,560 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32159.32 MB 2025-02-14 15:14:53,560 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13441.88 MB 2025-02-14 15:14:53,560 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52651.10 MB 2025-02-14 15:14:53,560 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37861.98 MB 2025-02-14 15:14:53,560 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14789.12 MB 2025-02-14 15:14:53,560 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32201.73 MB 2025-02-14 15:14:53,831 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 15:14:53,832 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 15:14:53,832 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 15:14:53,832 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:14:53,832 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32159.32 MB 2025-02-14 15:14:53,832 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23704.31 MB 2025-02-14 15:14:53,832 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8455.01 MB 2025-02-14 15:14:53,832 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37861.98 MB 2025-02-14 15:14:53,832 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37861.98 MB 2025-02-14 15:14:53,832 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:14:53,832 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34656.86 MB 2025-02-14 15:14:53,850 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8116, cut from 8118 2025-02-14 15:14:53,850 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-14 15:14:53,856 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 15:14:53,856 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 15:14:53,856 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:14:53,856 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:14:53,856 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23704.31 MB 2025-02-14 15:14:53,856 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32096.73 MB 2025-02-14 15:14:53,856 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8392.42 MB 2025-02-14 15:14:53,856 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37861.98 MB 2025-02-14 15:14:53,856 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46204.45 MB 2025-02-14 15:14:53,856 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8342.47 MB 2025-02-14 15:14:53,856 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32096.73 MB 2025-02-14 15:14:54,021 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7908] 2025-02-14 15:14:54,023 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:14:54,023 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:14:54,024 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:14:54,024 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 15:14:54,029 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 15:14:54,030 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:14:54,030 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 15:14:54,030 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-14 15:15:10,207 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:15:10,207 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:15:10,212 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 15:15:10,215 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:15:10,216 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1211, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 15:15:10,216 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:15:10,217 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1211, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 15:15:29,068 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 15:15:29,069 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 15:15:29,069 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.84 seconds 2025-02-14 15:15:29,069 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:15:29,069 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21407.15 MB 2025-02-14 15:15:29,069 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25693.73 MB 2025-02-14 15:15:29,069 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4286.58 MB 2025-02-14 15:15:29,069 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54546.92 MB 2025-02-14 15:15:29,069 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33707.52 MB 2025-02-14 15:15:29,069 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20839.40 MB 2025-02-14 15:15:29,069 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34502.40 MB 2025-02-14 15:15:29,146 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 15:15:29,146 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 15:15:29,146 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 15:15:29,146 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:15:29,146 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25693.73 MB 2025-02-14 15:15:29,146 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22073.46 MB 2025-02-14 15:15:29,146 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3620.26 MB 2025-02-14 15:15:29,147 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33707.52 MB 2025-02-14 15:15:29,147 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42580.57 MB 2025-02-14 15:15:29,147 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8873.05 MB 2025-02-14 15:15:29,147 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38265.68 MB 2025-02-14 15:15:31,094 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 15:15:31,094 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 15:15:31,094 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.95 seconds 2025-02-14 15:15:31,094 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:15:31,094 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22073.46 MB 2025-02-14 15:15:31,094 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22604.30 MB 2025-02-14 15:15:31,094 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 15:15:31,094 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42580.57 MB 2025-02-14 15:15:31,094 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25249.71 MB 2025-02-14 15:15:31,094 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17330.86 MB 2025-02-14 15:15:31,094 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26584.68 MB 2025-02-14 15:15:31,124 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 15:15:31,124 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 15:15:31,124 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 15:15:31,124 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:15:31,124 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22604.30 MB 2025-02-14 15:15:31,124 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24493.84 MB 2025-02-14 15:15:31,125 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 15:15:31,125 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25249.71 MB 2025-02-14 15:15:31,125 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28080.87 MB 2025-02-14 15:15:31,125 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-14 15:15:31,125 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25911.27 MB 2025-02-14 15:15:31,342 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 15:15:31,342 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 15:15:31,342 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 15:15:31,342 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:15:31,342 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24493.84 MB 2025-02-14 15:15:31,342 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26735.69 MB 2025-02-14 15:15:31,342 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 15:15:31,342 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28080.87 MB 2025-02-14 15:15:31,342 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34215.03 MB 2025-02-14 15:15:31,342 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 15:15:31,342 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32279.97 MB 2025-02-14 15:15:31,343 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 15:15:31,343 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 15:15:31,343 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.25 seconds 2025-02-14 15:15:31,343 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:15:31,343 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22604.30 MB 2025-02-14 15:15:31,343 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26735.69 MB 2025-02-14 15:15:31,343 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 15:15:31,343 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25249.71 MB 2025-02-14 15:15:31,343 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34215.03 MB 2025-02-14 15:15:31,343 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8965.32 MB 2025-02-14 15:15:31,343 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32279.97 MB 2025-02-14 15:15:31,523 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 15:15:31,523 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 15:15:31,523 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 15:15:31,523 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:15:31,523 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28269.24 MB 2025-02-14 15:15:31,523 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29036.24 MB 2025-02-14 15:15:31,523 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 15:15:31,523 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34215.03 MB 2025-02-14 15:15:31,523 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34632.37 MB 2025-02-14 15:15:31,523 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 15:15:31,523 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29744.03 MB 2025-02-14 15:15:31,544 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 15:15:31,544 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 15:15:31,544 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:15:31,544 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:15:31,544 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29449.13 MB 2025-02-14 15:15:31,544 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29676.15 MB 2025-02-14 15:15:31,544 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 227.02 MB 2025-02-14 15:15:31,544 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34632.37 MB 2025-02-14 15:15:31,544 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34632.37 MB 2025-02-14 15:15:31,544 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:15:31,544 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29902.76 MB 2025-02-14 15:15:31,546 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 15:15:31,546 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 15:15:31,546 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.33 seconds 2025-02-14 15:15:31,546 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:15:31,546 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17187.93 MB 2025-02-14 15:15:31,546 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29876.09 MB 2025-02-14 15:15:31,546 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12688.16 MB 2025-02-14 15:15:31,546 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54546.92 MB 2025-02-14 15:15:31,546 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34632.37 MB 2025-02-14 15:15:31,546 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19914.56 MB 2025-02-14 15:15:31,546 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29902.76 MB 2025-02-14 15:15:31,818 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 15:15:31,818 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 15:15:31,818 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 15:15:31,818 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:15:31,818 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29876.09 MB 2025-02-14 15:15:31,818 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22174.79 MB 2025-02-14 15:15:31,818 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7701.29 MB 2025-02-14 15:15:31,818 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34632.37 MB 2025-02-14 15:15:31,818 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34632.37 MB 2025-02-14 15:15:31,818 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:15:31,818 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32373.62 MB 2025-02-14 15:15:31,838 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8116, cut from 8118 2025-02-14 15:15:31,839 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 15:15:31,845 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 15:15:31,845 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 15:15:31,845 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 15:15:31,845 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:15:31,845 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22174.79 MB 2025-02-14 15:15:31,845 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30567.22 MB 2025-02-14 15:15:31,845 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8392.42 MB 2025-02-14 15:15:31,845 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34632.37 MB 2025-02-14 15:15:31,845 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42974.84 MB 2025-02-14 15:15:31,845 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8342.47 MB 2025-02-14 15:15:31,845 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30567.22 MB 2025-02-14 15:15:32,007 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7908] 2025-02-14 15:15:32,009 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:15:32,009 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:15:32,010 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:15:32,010 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 15:15:32,015 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 15:15:32,016 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:15:32,016 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 15:15:32,016 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 15:15:49,488 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:15:49,488 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:15:49,493 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 15:15:49,497 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:15:49,497 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 298, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 15:15:49,498 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:15:49,498 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 298, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 15:15:54,155 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 15:15:54,155 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 15:15:54,155 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.65 seconds 2025-02-14 15:15:54,155 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:15:54,155 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15045.22 MB 2025-02-14 15:15:54,155 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16099.82 MB 2025-02-14 15:15:54,155 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1054.61 MB 2025-02-14 15:15:54,155 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51317.31 MB 2025-02-14 15:15:54,155 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18796.77 MB 2025-02-14 15:15:54,155 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32520.54 MB 2025-02-14 15:15:54,156 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24970.38 MB 2025-02-14 15:15:54,177 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 15:15:54,177 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 15:15:54,177 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:15:54,177 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:15:54,177 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16099.82 MB 2025-02-14 15:15:54,177 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16527.58 MB 2025-02-14 15:15:54,177 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 427.75 MB 2025-02-14 15:15:54,177 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18796.77 MB 2025-02-14 15:15:54,177 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22819.11 MB 2025-02-14 15:15:54,177 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4022.34 MB 2025-02-14 15:15:54,177 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20154.67 MB 2025-02-14 15:15:55,559 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 15:15:55,559 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 15:15:55,559 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.38 seconds 2025-02-14 15:15:55,559 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:15:55,559 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16527.58 MB 2025-02-14 15:15:55,559 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16907.13 MB 2025-02-14 15:15:55,559 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 379.55 MB 2025-02-14 15:15:55,559 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22819.11 MB 2025-02-14 15:15:55,559 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20807.94 MB 2025-02-14 15:15:55,559 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2011.17 MB 2025-02-14 15:15:55,559 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20868.92 MB 2025-02-14 15:15:55,570 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 15:15:55,570 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 15:15:55,570 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:15:55,570 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:15:55,570 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16907.13 MB 2025-02-14 15:15:55,570 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18258.61 MB 2025-02-14 15:15:55,570 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1351.48 MB 2025-02-14 15:15:55,570 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20807.94 MB 2025-02-14 15:15:55,570 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21483.23 MB 2025-02-14 15:15:55,570 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 675.28 MB 2025-02-14 15:15:55,570 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19272.08 MB 2025-02-14 15:15:55,727 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 15:15:55,727 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 15:15:55,727 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 15:15:55,727 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:15:55,727 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18258.61 MB 2025-02-14 15:15:55,727 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19862.08 MB 2025-02-14 15:15:55,727 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1603.47 MB 2025-02-14 15:15:55,727 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21483.23 MB 2025-02-14 15:15:55,727 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25534.92 MB 2025-02-14 15:15:55,727 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4051.70 MB 2025-02-14 15:15:55,727 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23828.32 MB 2025-02-14 15:15:55,728 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 15:15:55,728 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 15:15:55,728 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 15:15:55,728 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:15:55,728 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16907.13 MB 2025-02-14 15:15:55,728 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19862.08 MB 2025-02-14 15:15:55,728 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2954.95 MB 2025-02-14 15:15:55,728 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20807.94 MB 2025-02-14 15:15:55,728 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25534.92 MB 2025-02-14 15:15:55,728 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4726.98 MB 2025-02-14 15:15:55,728 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23828.32 MB 2025-02-14 15:15:55,854 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 15:15:55,854 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 15:15:55,854 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 15:15:55,854 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:15:55,854 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20958.56 MB 2025-02-14 15:15:55,854 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21506.97 MB 2025-02-14 15:15:55,854 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 548.41 MB 2025-02-14 15:15:55,854 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25534.92 MB 2025-02-14 15:15:55,854 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25830.62 MB 2025-02-14 15:15:55,854 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 295.70 MB 2025-02-14 15:15:55,854 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22013.04 MB 2025-02-14 15:15:55,869 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 15:15:55,869 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 15:15:55,869 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:15:55,869 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:15:55,869 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21802.19 MB 2025-02-14 15:15:55,869 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22021.04 MB 2025-02-14 15:15:55,869 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 218.85 MB 2025-02-14 15:15:55,869 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25830.62 MB 2025-02-14 15:15:55,869 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25830.62 MB 2025-02-14 15:15:55,869 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:15:55,869 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22134.48 MB 2025-02-14 15:15:55,870 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 15:15:55,870 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 15:15:55,870 - resource_logging.py:150 - __exit__ - DEBUG - Time: 6.37 seconds 2025-02-14 15:15:55,870 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:15:55,870 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14006.96 MB 2025-02-14 15:15:55,870 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22221.52 MB 2025-02-14 15:15:55,870 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8214.56 MB 2025-02-14 15:15:55,870 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51317.31 MB 2025-02-14 15:15:55,870 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25830.62 MB 2025-02-14 15:15:55,870 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25486.69 MB 2025-02-14 15:15:55,870 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22221.52 MB 2025-02-14 15:15:56,142 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 15:15:56,142 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 15:15:56,142 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 15:15:56,142 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:15:56,142 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22221.52 MB 2025-02-14 15:15:56,142 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25226.71 MB 2025-02-14 15:15:56,142 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3005.19 MB 2025-02-14 15:15:56,142 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25830.62 MB 2025-02-14 15:15:56,142 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26770.15 MB 2025-02-14 15:15:56,142 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 939.52 MB 2025-02-14 15:15:56,142 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25527.85 MB 2025-02-14 15:15:56,160 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8138, cut from 8140 2025-02-14 15:15:56,160 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-14 15:15:56,166 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 15:15:56,166 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 15:15:56,166 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:15:56,166 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:15:56,166 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18464.21 MB 2025-02-14 15:15:56,166 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26878.19 MB 2025-02-14 15:15:56,166 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8413.98 MB 2025-02-14 15:15:56,166 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26770.15 MB 2025-02-14 15:15:56,166 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37228.64 MB 2025-02-14 15:15:56,166 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10458.50 MB 2025-02-14 15:15:56,166 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26878.19 MB 2025-02-14 15:15:56,331 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7930] 2025-02-14 15:15:56,332 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:15:56,332 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:15:56,333 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:15:56,333 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 15:15:56,338 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 15:15:56,339 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:15:56,339 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 15:15:56,339 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-14 15:16:05,410 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:16:05,410 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:16:05,415 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 15:16:05,419 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:16:05,419 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 332, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 15:16:05,420 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:16:05,420 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 332, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 15:16:10,630 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 15:16:10,630 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 15:16:10,630 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.21 seconds 2025-02-14 15:16:10,630 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:16:10,630 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15282.14 MB 2025-02-14 15:16:10,630 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16457.07 MB 2025-02-14 15:16:10,630 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1174.93 MB 2025-02-14 15:16:10,630 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49775.90 MB 2025-02-14 15:16:10,630 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20109.59 MB 2025-02-14 15:16:10,630 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29666.31 MB 2025-02-14 15:16:10,630 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25433.79 MB 2025-02-14 15:16:10,654 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 15:16:10,654 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 15:16:10,654 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:16:10,654 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:16:10,654 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16457.07 MB 2025-02-14 15:16:10,654 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17026.25 MB 2025-02-14 15:16:10,654 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 569.18 MB 2025-02-14 15:16:10,654 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20109.59 MB 2025-02-14 15:16:10,654 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23033.02 MB 2025-02-14 15:16:10,654 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2923.43 MB 2025-02-14 15:16:10,654 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21162.82 MB 2025-02-14 15:16:12,285 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 15:16:12,285 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 15:16:12,285 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.63 seconds 2025-02-14 15:16:12,285 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:16:12,285 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17026.25 MB 2025-02-14 15:16:12,285 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17466.85 MB 2025-02-14 15:16:12,285 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 440.60 MB 2025-02-14 15:16:12,285 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23033.02 MB 2025-02-14 15:16:12,285 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19434.31 MB 2025-02-14 15:16:12,285 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3598.71 MB 2025-02-14 15:16:12,285 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21452.53 MB 2025-02-14 15:16:12,297 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 15:16:12,297 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 15:16:12,297 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:16:12,297 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:16:12,297 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17466.85 MB 2025-02-14 15:16:12,297 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19037.09 MB 2025-02-14 15:16:12,297 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1570.24 MB 2025-02-14 15:16:12,297 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19434.31 MB 2025-02-14 15:16:12,297 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21787.31 MB 2025-02-14 15:16:12,297 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2353.00 MB 2025-02-14 15:16:12,297 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20214.61 MB 2025-02-14 15:16:12,475 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 15:16:12,475 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 15:16:12,475 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-14 15:16:12,475 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:16:12,475 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19037.09 MB 2025-02-14 15:16:12,476 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20898.89 MB 2025-02-14 15:16:12,476 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1861.80 MB 2025-02-14 15:16:12,476 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21787.31 MB 2025-02-14 15:16:12,476 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27082.62 MB 2025-02-14 15:16:12,476 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5295.31 MB 2025-02-14 15:16:12,476 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25506.40 MB 2025-02-14 15:16:12,476 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 15:16:12,476 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 15:16:12,476 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.19 seconds 2025-02-14 15:16:12,476 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:16:12,476 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17466.85 MB 2025-02-14 15:16:12,476 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20898.89 MB 2025-02-14 15:16:12,476 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3432.04 MB 2025-02-14 15:16:12,476 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19434.31 MB 2025-02-14 15:16:12,476 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27082.62 MB 2025-02-14 15:16:12,476 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7648.31 MB 2025-02-14 15:16:12,476 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25506.40 MB 2025-02-14 15:16:12,624 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 15:16:12,624 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 15:16:12,624 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-14 15:16:12,624 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:16:12,624 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22171.73 MB 2025-02-14 15:16:12,624 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22809.39 MB 2025-02-14 15:16:12,624 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 637.66 MB 2025-02-14 15:16:12,624 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27082.62 MB 2025-02-14 15:16:12,624 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27428.65 MB 2025-02-14 15:16:12,624 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 346.03 MB 2025-02-14 15:16:12,624 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23396.86 MB 2025-02-14 15:16:12,641 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 15:16:12,641 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 15:16:12,641 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:16:12,641 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:16:12,641 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23152.09 MB 2025-02-14 15:16:12,641 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23365.41 MB 2025-02-14 15:16:12,641 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 213.32 MB 2025-02-14 15:16:12,641 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27428.65 MB 2025-02-14 15:16:12,641 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27428.65 MB 2025-02-14 15:16:12,641 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:16:12,641 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23510.79 MB 2025-02-14 15:16:12,642 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 15:16:12,642 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 15:16:12,642 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.22 seconds 2025-02-14 15:16:12,642 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:16:12,642 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14125.42 MB 2025-02-14 15:16:12,642 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23566.48 MB 2025-02-14 15:16:12,642 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 9441.06 MB 2025-02-14 15:16:12,642 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49775.90 MB 2025-02-14 15:16:12,642 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27428.65 MB 2025-02-14 15:16:12,643 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22347.25 MB 2025-02-14 15:16:12,643 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23566.48 MB 2025-02-14 15:16:12,912 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 15:16:12,912 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 15:16:12,912 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 15:16:12,912 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:16:12,912 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23566.48 MB 2025-02-14 15:16:12,912 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26580.52 MB 2025-02-14 15:16:12,912 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3014.03 MB 2025-02-14 15:16:12,912 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27428.65 MB 2025-02-14 15:16:12,912 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27965.52 MB 2025-02-14 15:16:12,912 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 536.87 MB 2025-02-14 15:16:12,912 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26882.15 MB 2025-02-14 15:16:12,930 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 15:16:12,930 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-14 15:16:12,936 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 15:16:12,936 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 15:16:12,936 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:16:12,936 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:16:12,936 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18809.95 MB 2025-02-14 15:16:12,936 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27248.97 MB 2025-02-14 15:16:12,936 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8439.02 MB 2025-02-14 15:16:12,936 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27965.52 MB 2025-02-14 15:16:12,936 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38455.48 MB 2025-02-14 15:16:12,936 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 15:16:12,936 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27248.97 MB 2025-02-14 15:16:13,099 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 15:16:13,100 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:16:13,100 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:16:13,101 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:16:13,101 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 15:16:13,106 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 15:16:13,107 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:16:13,107 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 15:16:13,107 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-14 15:17:03,635 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:17:03,635 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:17:03,640 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 15:17:03,644 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:17:03,644 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 172, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 15:17:03,645 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:17:03,645 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 172, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 15:17:06,323 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 15:17:06,323 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 15:17:06,323 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.67 seconds 2025-02-14 15:17:06,323 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:17:06,323 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14167.23 MB 2025-02-14 15:17:06,323 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14775.93 MB 2025-02-14 15:17:06,323 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 608.70 MB 2025-02-14 15:17:06,323 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51040.49 MB 2025-02-14 15:17:06,323 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18475.91 MB 2025-02-14 15:17:06,323 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32564.58 MB 2025-02-14 15:17:06,323 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23639.41 MB 2025-02-14 15:17:06,336 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 15:17:06,336 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 15:17:06,336 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:17:06,336 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:17:06,336 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14775.93 MB 2025-02-14 15:17:06,336 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14993.59 MB 2025-02-14 15:17:06,336 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 217.66 MB 2025-02-14 15:17:06,336 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18475.91 MB 2025-02-14 15:17:06,336 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18475.91 MB 2025-02-14 15:17:06,336 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:17:06,336 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17079.87 MB 2025-02-14 15:17:07,121 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 15:17:07,122 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 15:17:07,122 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.78 seconds 2025-02-14 15:17:07,122 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:17:07,122 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14993.59 MB 2025-02-14 15:17:07,122 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15207.25 MB 2025-02-14 15:17:07,122 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 213.66 MB 2025-02-14 15:17:07,122 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18475.91 MB 2025-02-14 15:17:07,122 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18083.74 MB 2025-02-14 15:17:07,122 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -392.17 MB 2025-02-14 15:17:07,122 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19165.06 MB 2025-02-14 15:17:07,133 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 15:17:07,133 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 15:17:07,133 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:17:07,133 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:17:07,133 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15207.19 MB 2025-02-14 15:17:07,133 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15967.54 MB 2025-02-14 15:17:07,134 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 760.35 MB 2025-02-14 15:17:07,134 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18083.74 MB 2025-02-14 15:17:07,134 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18465.42 MB 2025-02-14 15:17:07,134 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 381.68 MB 2025-02-14 15:17:07,134 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16538.06 MB 2025-02-14 15:17:07,309 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 15:17:07,309 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 15:17:07,309 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 15:17:07,309 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:17:07,309 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15967.54 MB 2025-02-14 15:17:07,309 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16869.93 MB 2025-02-14 15:17:07,309 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 902.39 MB 2025-02-14 15:17:07,309 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18465.42 MB 2025-02-14 15:17:07,309 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20564.67 MB 2025-02-14 15:17:07,309 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2099.25 MB 2025-02-14 15:17:07,309 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19102.38 MB 2025-02-14 15:17:07,310 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 15:17:07,310 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 15:17:07,310 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.19 seconds 2025-02-14 15:17:07,310 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:17:07,310 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15207.19 MB 2025-02-14 15:17:07,310 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16869.93 MB 2025-02-14 15:17:07,310 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1662.74 MB 2025-02-14 15:17:07,310 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18083.74 MB 2025-02-14 15:17:07,310 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20564.67 MB 2025-02-14 15:17:07,310 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2480.93 MB 2025-02-14 15:17:07,310 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19102.38 MB 2025-02-14 15:17:07,458 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 15:17:07,458 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 15:17:07,459 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-14 15:17:07,459 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:17:07,459 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17487.18 MB 2025-02-14 15:17:07,459 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17796.81 MB 2025-02-14 15:17:07,459 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 309.64 MB 2025-02-14 15:17:07,459 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20564.67 MB 2025-02-14 15:17:07,459 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20730.35 MB 2025-02-14 15:17:07,459 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 165.68 MB 2025-02-14 15:17:07,459 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18091.13 MB 2025-02-14 15:17:07,469 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 15:17:07,469 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 15:17:07,469 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:17:07,469 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:17:07,469 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17963.01 MB 2025-02-14 15:17:07,469 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18191.47 MB 2025-02-14 15:17:07,469 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.46 MB 2025-02-14 15:17:07,469 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20730.35 MB 2025-02-14 15:17:07,469 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20730.35 MB 2025-02-14 15:17:07,469 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:17:07,469 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18206.94 MB 2025-02-14 15:17:07,470 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 15:17:07,470 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 15:17:07,470 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.82 seconds 2025-02-14 15:17:07,470 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:17:07,470 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13567.97 MB 2025-02-14 15:17:07,470 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18392.32 MB 2025-02-14 15:17:07,470 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4824.35 MB 2025-02-14 15:17:07,470 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51040.49 MB 2025-02-14 15:17:07,471 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20730.35 MB 2025-02-14 15:17:07,471 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30310.14 MB 2025-02-14 15:17:07,471 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18392.32 MB 2025-02-14 15:17:07,740 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 15:17:07,740 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 15:17:07,740 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 15:17:07,740 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:17:07,740 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18392.32 MB 2025-02-14 15:17:07,740 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17441.67 MB 2025-02-14 15:17:07,740 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -950.64 MB 2025-02-14 15:17:07,740 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20730.35 MB 2025-02-14 15:17:07,740 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20730.35 MB 2025-02-14 15:17:07,740 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:17:07,740 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19195.17 MB 2025-02-14 15:17:07,758 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8153, cut from 8155 2025-02-14 15:17:07,759 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 15:17:07,766 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 15:17:07,766 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 15:17:07,766 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:17:07,767 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:17:07,767 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17441.67 MB 2025-02-14 15:17:07,767 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25872.07 MB 2025-02-14 15:17:07,767 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8430.40 MB 2025-02-14 15:17:07,767 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20730.35 MB 2025-02-14 15:17:07,767 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31205.62 MB 2025-02-14 15:17:07,767 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10475.27 MB 2025-02-14 15:17:07,767 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25872.07 MB 2025-02-14 15:17:07,918 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7945] 2025-02-14 15:17:07,920 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:17:07,920 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:17:07,921 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:17:07,921 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 15:17:07,925 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 15:17:07,926 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:17:07,926 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 15:17:07,926 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 15:19:05,035 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:19:05,035 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:19:05,041 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 15:19:05,045 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:19:05,046 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1147, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 15:19:05,047 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:19:05,048 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1147, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 15:19:22,593 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 15:19:22,593 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 15:19:22,593 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.54 seconds 2025-02-14 15:19:22,593 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:19:22,593 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20961.19 MB 2025-02-14 15:19:22,593 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25021.27 MB 2025-02-14 15:19:22,593 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4060.09 MB 2025-02-14 15:19:22,593 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39585.84 MB 2025-02-14 15:19:22,593 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31444.70 MB 2025-02-14 15:19:22,593 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8141.14 MB 2025-02-14 15:19:22,593 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33829.94 MB 2025-02-14 15:19:22,667 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 15:19:22,667 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 15:19:22,667 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 15:19:22,667 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:19:22,667 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25021.27 MB 2025-02-14 15:19:22,667 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21740.75 MB 2025-02-14 15:19:22,667 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3280.53 MB 2025-02-14 15:19:22,667 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31444.70 MB 2025-02-14 15:19:22,667 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42054.19 MB 2025-02-14 15:19:22,667 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10609.49 MB 2025-02-14 15:19:22,667 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36975.79 MB 2025-02-14 15:19:24,583 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 15:19:24,583 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 15:19:24,583 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 15:19:24,583 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:19:24,583 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21740.75 MB 2025-02-14 15:19:24,583 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22271.59 MB 2025-02-14 15:19:24,583 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 15:19:24,583 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42054.19 MB 2025-02-14 15:19:24,583 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28800.19 MB 2025-02-14 15:19:24,583 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13254.00 MB 2025-02-14 15:19:24,583 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26250.92 MB 2025-02-14 15:19:24,596 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 15:19:24,596 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 15:19:24,596 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:19:24,596 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:19:24,597 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22271.59 MB 2025-02-14 15:19:24,597 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24161.12 MB 2025-02-14 15:19:24,597 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 15:19:24,597 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28800.19 MB 2025-02-14 15:19:24,597 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28800.19 MB 2025-02-14 15:19:24,597 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:19:24,597 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25578.55 MB 2025-02-14 15:19:24,822 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 15:19:24,822 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 15:19:24,822 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 15:19:24,822 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:19:24,822 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24161.12 MB 2025-02-14 15:19:24,822 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26402.98 MB 2025-02-14 15:19:24,822 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 15:19:24,822 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28800.19 MB 2025-02-14 15:19:24,822 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34462.50 MB 2025-02-14 15:19:24,822 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 15:19:24,822 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31947.26 MB 2025-02-14 15:19:24,823 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 15:19:24,823 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 15:19:24,823 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.24 seconds 2025-02-14 15:19:24,823 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:19:24,823 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22271.59 MB 2025-02-14 15:19:24,823 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26402.98 MB 2025-02-14 15:19:24,823 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 15:19:24,823 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28800.19 MB 2025-02-14 15:19:24,823 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34462.50 MB 2025-02-14 15:19:24,823 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 15:19:24,823 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31947.26 MB 2025-02-14 15:19:24,997 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 15:19:24,997 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 15:19:24,997 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 15:19:24,997 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:19:24,997 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27936.52 MB 2025-02-14 15:19:24,997 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28703.52 MB 2025-02-14 15:19:24,997 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 15:19:24,997 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34462.50 MB 2025-02-14 15:19:24,997 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34877.73 MB 2025-02-14 15:19:24,997 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 15:19:24,997 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29411.31 MB 2025-02-14 15:19:25,017 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 15:19:25,017 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 15:19:25,017 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:19:25,017 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:19:25,017 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29116.41 MB 2025-02-14 15:19:25,017 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29344.65 MB 2025-02-14 15:19:25,017 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.24 MB 2025-02-14 15:19:25,017 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34877.73 MB 2025-02-14 15:19:25,017 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34877.73 MB 2025-02-14 15:19:25,017 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:19:25,017 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29552.03 MB 2025-02-14 15:19:25,018 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 15:19:25,018 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 15:19:25,018 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.97 seconds 2025-02-14 15:19:25,018 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:19:25,018 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16964.95 MB 2025-02-14 15:19:25,018 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29545.13 MB 2025-02-14 15:19:25,018 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12580.18 MB 2025-02-14 15:19:25,018 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39585.84 MB 2025-02-14 15:19:25,018 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34877.73 MB 2025-02-14 15:19:25,018 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4708.11 MB 2025-02-14 15:19:25,018 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29552.03 MB 2025-02-14 15:19:25,285 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 15:19:25,285 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 15:19:25,285 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 15:19:25,285 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:19:25,285 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29545.13 MB 2025-02-14 15:19:25,286 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21960.19 MB 2025-02-14 15:19:25,286 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7584.94 MB 2025-02-14 15:19:25,286 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34877.73 MB 2025-02-14 15:19:25,286 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34877.73 MB 2025-02-14 15:19:25,286 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:19:25,286 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32049.42 MB 2025-02-14 15:19:25,303 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8138, cut from 8140 2025-02-14 15:19:25,303 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 15:19:25,309 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 15:19:25,309 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 15:19:25,309 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:19:25,309 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:19:25,309 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21960.19 MB 2025-02-14 15:19:25,309 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30374.11 MB 2025-02-14 15:19:25,309 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8413.92 MB 2025-02-14 15:19:25,309 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34877.73 MB 2025-02-14 15:19:25,309 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39061.55 MB 2025-02-14 15:19:25,309 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4183.82 MB 2025-02-14 15:19:25,309 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30374.11 MB 2025-02-14 15:19:25,473 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7930] 2025-02-14 15:19:25,474 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:19:25,474 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:19:25,475 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:19:25,475 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 15:19:25,480 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 15:19:25,481 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:19:25,481 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 15:19:25,481 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 15:19:34,058 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:19:34,058 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:19:34,063 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 15:19:34,066 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:19:34,066 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2369, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 15:19:34,067 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:19:34,067 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2369, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 15:20:10,944 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 15:20:10,945 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 15:20:10,945 - resource_logging.py:150 - __exit__ - DEBUG - Time: 36.87 seconds 2025-02-14 15:20:10,945 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:20:10,945 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29476.93 MB 2025-02-14 15:20:10,945 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37861.35 MB 2025-02-14 15:20:10,945 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8384.41 MB 2025-02-14 15:20:10,945 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55809.41 MB 2025-02-14 15:20:10,945 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42041.61 MB 2025-02-14 15:20:10,945 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13767.80 MB 2025-02-14 15:20:10,945 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46876.34 MB 2025-02-14 15:20:11,098 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 15:20:11,098 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 15:20:11,098 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 15:20:11,098 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:20:11,098 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37861.35 MB 2025-02-14 15:20:11,098 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28093.87 MB 2025-02-14 15:20:11,098 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9767.48 MB 2025-02-14 15:20:11,098 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42041.61 MB 2025-02-14 15:20:11,098 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 69839.36 MB 2025-02-14 15:20:11,098 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 27797.75 MB 2025-02-14 15:20:11,098 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 59421.25 MB 2025-02-14 15:20:13,083 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 15:20:13,083 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 15:20:13,083 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.98 seconds 2025-02-14 15:20:13,084 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:20:13,084 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28093.87 MB 2025-02-14 15:20:13,084 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28624.71 MB 2025-02-14 15:20:13,084 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 15:20:13,084 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 69839.36 MB 2025-02-14 15:20:13,084 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30880.56 MB 2025-02-14 15:20:13,084 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -38958.79 MB 2025-02-14 15:20:13,084 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32605.08 MB 2025-02-14 15:20:13,097 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 15:20:13,097 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 15:20:13,097 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:20:13,097 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:20:13,097 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28624.71 MB 2025-02-14 15:20:13,097 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30514.25 MB 2025-02-14 15:20:13,098 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 15:20:13,098 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30880.56 MB 2025-02-14 15:20:13,098 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34183.58 MB 2025-02-14 15:20:13,098 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 15:20:13,098 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31931.67 MB 2025-02-14 15:20:13,307 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 15:20:13,307 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 15:20:13,307 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 15:20:13,307 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:20:13,307 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30514.25 MB 2025-02-14 15:20:13,307 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32756.10 MB 2025-02-14 15:20:13,307 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 15:20:13,307 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34183.58 MB 2025-02-14 15:20:13,307 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40789.61 MB 2025-02-14 15:20:13,307 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 15:20:13,307 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38300.38 MB 2025-02-14 15:20:13,308 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 15:20:13,308 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 15:20:13,308 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 15:20:13,308 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:20:13,308 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28624.71 MB 2025-02-14 15:20:13,308 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32756.10 MB 2025-02-14 15:20:13,308 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 15:20:13,308 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30880.56 MB 2025-02-14 15:20:13,308 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40789.61 MB 2025-02-14 15:20:13,308 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9909.04 MB 2025-02-14 15:20:13,308 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38300.38 MB 2025-02-14 15:20:13,480 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 15:20:13,480 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 15:20:13,480 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 15:20:13,480 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:20:13,480 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34289.64 MB 2025-02-14 15:20:13,480 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35056.65 MB 2025-02-14 15:20:13,480 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 15:20:13,480 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40789.61 MB 2025-02-14 15:20:13,480 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41204.84 MB 2025-02-14 15:20:13,480 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 15:20:13,480 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35764.43 MB 2025-02-14 15:20:13,500 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 15:20:13,500 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 15:20:13,500 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:20:13,500 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:20:13,500 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35469.53 MB 2025-02-14 15:20:13,500 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35697.54 MB 2025-02-14 15:20:13,500 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.00 MB 2025-02-14 15:20:13,500 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41204.84 MB 2025-02-14 15:20:13,500 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41204.84 MB 2025-02-14 15:20:13,500 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:20:13,500 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35922.78 MB 2025-02-14 15:20:13,501 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 15:20:13,501 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 15:20:13,501 - resource_logging.py:150 - __exit__ - DEBUG - Time: 39.43 seconds 2025-02-14 15:20:13,501 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:20:13,501 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21222.82 MB 2025-02-14 15:20:13,501 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35897.70 MB 2025-02-14 15:20:13,501 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14674.88 MB 2025-02-14 15:20:13,501 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51617.20 MB 2025-02-14 15:20:13,501 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41204.84 MB 2025-02-14 15:20:13,501 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10412.36 MB 2025-02-14 15:20:13,501 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35922.78 MB 2025-02-14 15:20:13,802 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 15:20:13,802 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 15:20:13,803 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.30 seconds 2025-02-14 15:20:13,803 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:20:13,803 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35897.70 MB 2025-02-14 15:20:13,803 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26212.87 MB 2025-02-14 15:20:13,803 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9684.83 MB 2025-02-14 15:20:13,803 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41204.84 MB 2025-02-14 15:20:13,803 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41204.84 MB 2025-02-14 15:20:13,803 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:20:13,803 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38398.00 MB 2025-02-14 15:20:13,821 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8125, cut from 8127 2025-02-14 15:20:13,821 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 15:20:13,827 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 15:20:13,827 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 15:20:13,827 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:20:13,827 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:20:13,827 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26212.87 MB 2025-02-14 15:20:13,827 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34613.81 MB 2025-02-14 15:20:13,827 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8400.94 MB 2025-02-14 15:20:13,827 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41204.84 MB 2025-02-14 15:20:13,827 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49557.80 MB 2025-02-14 15:20:13,827 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8352.96 MB 2025-02-14 15:20:13,827 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34613.81 MB 2025-02-14 15:20:13,989 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7917] 2025-02-14 15:20:13,990 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:20:13,990 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:20:13,991 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:20:13,991 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 15:20:13,996 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 15:20:13,997 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:20:13,997 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 15:20:13,997 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 15:21:09,929 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:21:09,930 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:21:09,935 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 15:21:09,938 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:21:09,938 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 156, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 15:21:09,939 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:21:09,939 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 156, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 15:21:12,364 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 15:21:12,364 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 15:21:12,365 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.42 seconds 2025-02-14 15:21:12,365 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:21:12,365 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14055.74 MB 2025-02-14 15:21:12,365 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14607.81 MB 2025-02-14 15:21:12,365 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 552.08 MB 2025-02-14 15:21:12,365 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62086.18 MB 2025-02-14 15:21:12,365 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18324.91 MB 2025-02-14 15:21:12,365 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -43761.27 MB 2025-02-14 15:21:12,365 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23527.92 MB 2025-02-14 15:21:12,377 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 15:21:12,377 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 15:21:12,377 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:21:12,377 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:21:12,377 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14607.81 MB 2025-02-14 15:21:12,377 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 14875.29 MB 2025-02-14 15:21:12,377 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 267.48 MB 2025-02-14 15:21:12,377 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18324.91 MB 2025-02-14 15:21:12,377 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18324.91 MB 2025-02-14 15:21:12,377 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:21:12,377 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16841.53 MB 2025-02-14 15:21:13,131 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 15:21:13,131 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 15:21:13,131 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.75 seconds 2025-02-14 15:21:13,131 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:21:13,131 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 14875.29 MB 2025-02-14 15:21:13,131 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15082.39 MB 2025-02-14 15:21:13,131 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 207.09 MB 2025-02-14 15:21:13,131 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18324.91 MB 2025-02-14 15:21:13,131 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18324.91 MB 2025-02-14 15:21:13,131 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:21:13,131 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19046.77 MB 2025-02-14 15:21:13,138 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 15:21:13,139 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 15:21:13,139 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 15:21:13,139 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:21:13,139 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15082.32 MB 2025-02-14 15:21:13,139 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 15819.06 MB 2025-02-14 15:21:13,139 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 736.74 MB 2025-02-14 15:21:13,139 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18324.91 MB 2025-02-14 15:21:13,139 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 18324.91 MB 2025-02-14 15:21:13,139 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:21:13,139 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 16371.86 MB 2025-02-14 15:21:13,228 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 15:21:13,228 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 15:21:13,228 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 15:21:13,228 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:21:13,228 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15819.06 MB 2025-02-14 15:21:13,228 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16693.42 MB 2025-02-14 15:21:13,228 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 874.36 MB 2025-02-14 15:21:13,228 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18324.91 MB 2025-02-14 15:21:13,228 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19801.31 MB 2025-02-14 15:21:13,228 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1476.40 MB 2025-02-14 15:21:13,228 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18859.85 MB 2025-02-14 15:21:13,228 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 15:21:13,228 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 15:21:13,228 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 15:21:13,228 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:21:13,228 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15082.32 MB 2025-02-14 15:21:13,228 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 16693.42 MB 2025-02-14 15:21:13,229 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1611.10 MB 2025-02-14 15:21:13,229 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 18324.91 MB 2025-02-14 15:21:13,229 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19801.31 MB 2025-02-14 15:21:13,229 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1476.40 MB 2025-02-14 15:21:13,229 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18859.85 MB 2025-02-14 15:21:13,296 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 15:21:13,296 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 15:21:13,296 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 15:21:13,297 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:21:13,297 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17291.51 MB 2025-02-14 15:21:13,297 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17590.64 MB 2025-02-14 15:21:13,297 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 299.13 MB 2025-02-14 15:21:13,297 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19801.31 MB 2025-02-14 15:21:13,297 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19960.69 MB 2025-02-14 15:21:13,297 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 159.38 MB 2025-02-14 15:21:13,297 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17874.63 MB 2025-02-14 15:21:13,306 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 15:21:13,306 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 15:21:13,306 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:21:13,306 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:21:13,306 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17751.67 MB 2025-02-14 15:21:13,306 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17963.42 MB 2025-02-14 15:21:13,306 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 211.75 MB 2025-02-14 15:21:13,306 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19960.69 MB 2025-02-14 15:21:13,306 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19960.69 MB 2025-02-14 15:21:13,306 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:21:13,306 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17980.65 MB 2025-02-14 15:21:13,307 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 15:21:13,307 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 15:21:13,307 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.37 seconds 2025-02-14 15:21:13,307 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:21:13,307 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 13512.22 MB 2025-02-14 15:21:13,307 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18164.25 MB 2025-02-14 15:21:13,307 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4652.03 MB 2025-02-14 15:21:13,307 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62086.18 MB 2025-02-14 15:21:13,307 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19960.69 MB 2025-02-14 15:21:13,307 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -42125.49 MB 2025-02-14 15:21:13,307 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18164.25 MB 2025-02-14 15:21:13,573 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 15:21:13,573 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 15:21:13,574 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 15:21:13,574 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:21:13,574 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18164.25 MB 2025-02-14 15:21:13,574 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17361.29 MB 2025-02-14 15:21:13,574 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -802.95 MB 2025-02-14 15:21:13,574 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19960.69 MB 2025-02-14 15:21:13,574 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19960.69 MB 2025-02-14 15:21:13,574 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:21:13,574 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19067.34 MB 2025-02-14 15:21:13,591 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8152, cut from 8154 2025-02-14 15:21:13,592 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-14 15:21:13,598 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 15:21:13,598 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 15:21:13,598 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:21:13,598 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:21:13,598 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17361.29 MB 2025-02-14 15:21:13,598 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25790.42 MB 2025-02-14 15:21:13,598 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8429.12 MB 2025-02-14 15:21:13,598 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19960.69 MB 2025-02-14 15:21:13,598 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30435.97 MB 2025-02-14 15:21:13,598 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10475.27 MB 2025-02-14 15:21:13,598 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25790.42 MB 2025-02-14 15:21:13,763 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7944] 2025-02-14 15:21:13,764 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:21:13,764 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:21:13,765 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:21:13,765 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 15:21:13,770 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 15:21:13,771 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:21:13,771 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 15:21:13,771 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-14 15:22:50,164 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:22:50,164 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:22:50,168 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 15:22:50,172 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:22:50,172 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1177, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 15:22:50,173 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:22:50,173 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1177, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 15:23:08,185 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 15:23:08,185 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 15:23:08,186 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.01 seconds 2025-02-14 15:23:08,186 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:23:08,186 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21170.23 MB 2025-02-14 15:23:08,186 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25335.57 MB 2025-02-14 15:23:08,186 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4165.34 MB 2025-02-14 15:23:08,186 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38816.19 MB 2025-02-14 15:23:08,186 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31551.65 MB 2025-02-14 15:23:08,186 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7264.53 MB 2025-02-14 15:23:08,186 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34265.48 MB 2025-02-14 15:23:08,262 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 15:23:08,262 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 15:23:08,262 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 15:23:08,262 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:23:08,262 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25335.57 MB 2025-02-14 15:23:08,262 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21896.71 MB 2025-02-14 15:23:08,262 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3438.86 MB 2025-02-14 15:23:08,262 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31551.65 MB 2025-02-14 15:23:08,262 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42819.65 MB 2025-02-14 15:23:08,262 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 11268.00 MB 2025-02-14 15:23:08,262 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37507.74 MB 2025-02-14 15:23:10,172 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 15:23:10,172 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 15:23:10,172 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 15:23:10,172 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:23:10,172 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21896.71 MB 2025-02-14 15:23:10,172 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22427.55 MB 2025-02-14 15:23:10,172 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 15:23:10,172 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42819.65 MB 2025-02-14 15:23:10,172 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28800.19 MB 2025-02-14 15:23:10,172 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14019.46 MB 2025-02-14 15:23:10,172 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26406.88 MB 2025-02-14 15:23:10,186 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 15:23:10,186 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 15:23:10,186 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:23:10,186 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:23:10,186 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22427.55 MB 2025-02-14 15:23:10,186 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24317.08 MB 2025-02-14 15:23:10,186 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 15:23:10,186 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28800.19 MB 2025-02-14 15:23:10,186 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28800.19 MB 2025-02-14 15:23:10,186 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:23:10,186 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25734.51 MB 2025-02-14 15:23:10,403 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 15:23:10,403 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 15:23:10,403 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 15:23:10,403 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:23:10,403 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24317.08 MB 2025-02-14 15:23:10,403 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26558.94 MB 2025-02-14 15:23:10,403 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 15:23:10,403 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28800.19 MB 2025-02-14 15:23:10,403 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34462.50 MB 2025-02-14 15:23:10,403 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 15:23:10,403 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32103.22 MB 2025-02-14 15:23:10,404 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 15:23:10,404 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 15:23:10,404 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 15:23:10,404 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:23:10,404 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22427.55 MB 2025-02-14 15:23:10,404 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26558.94 MB 2025-02-14 15:23:10,404 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 15:23:10,404 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28800.19 MB 2025-02-14 15:23:10,404 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34462.50 MB 2025-02-14 15:23:10,404 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 15:23:10,404 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32103.22 MB 2025-02-14 15:23:10,578 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 15:23:10,578 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 15:23:10,578 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 15:23:10,578 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:23:10,578 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28092.48 MB 2025-02-14 15:23:10,578 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28859.48 MB 2025-02-14 15:23:10,578 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 15:23:10,578 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34462.50 MB 2025-02-14 15:23:10,578 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34877.73 MB 2025-02-14 15:23:10,578 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 15:23:10,578 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29567.27 MB 2025-02-14 15:23:10,598 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 15:23:10,598 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 15:23:10,598 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:23:10,598 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:23:10,598 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29272.37 MB 2025-02-14 15:23:10,598 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29501.30 MB 2025-02-14 15:23:10,598 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.92 MB 2025-02-14 15:23:10,598 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34877.73 MB 2025-02-14 15:23:10,598 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34877.73 MB 2025-02-14 15:23:10,598 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:23:10,598 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29699.93 MB 2025-02-14 15:23:10,599 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 15:23:10,599 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 15:23:10,599 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.42 seconds 2025-02-14 15:23:10,599 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:23:10,599 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17069.47 MB 2025-02-14 15:23:10,599 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29701.73 MB 2025-02-14 15:23:10,599 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12632.26 MB 2025-02-14 15:23:10,599 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38816.19 MB 2025-02-14 15:23:10,599 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34877.73 MB 2025-02-14 15:23:10,599 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3938.45 MB 2025-02-14 15:23:10,599 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29701.73 MB 2025-02-14 15:23:10,867 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 15:23:10,867 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 15:23:10,867 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 15:23:10,867 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:23:10,867 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29701.73 MB 2025-02-14 15:23:10,867 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22063.95 MB 2025-02-14 15:23:10,867 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7637.78 MB 2025-02-14 15:23:10,867 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34877.73 MB 2025-02-14 15:23:10,867 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34877.73 MB 2025-02-14 15:23:10,867 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:23:10,867 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32205.41 MB 2025-02-14 15:23:10,885 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8136, cut from 8138 2025-02-14 15:23:10,885 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 1 ('] 2025-02-14 15:23:10,891 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 15:23:10,891 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 15:23:10,891 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:23:10,891 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:23:10,891 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22063.95 MB 2025-02-14 15:23:10,891 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30476.38 MB 2025-02-14 15:23:10,891 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8412.43 MB 2025-02-14 15:23:10,891 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34877.73 MB 2025-02-14 15:23:10,891 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43241.18 MB 2025-02-14 15:23:10,891 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8363.44 MB 2025-02-14 15:23:10,891 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30476.38 MB 2025-02-14 15:23:11,059 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7928] 2025-02-14 15:23:11,060 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:23:11,060 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:23:11,061 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:23:11,061 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 15:23:11,066 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 15:23:11,067 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:23:11,067 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 15:23:11,067 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 1 ('] 2025-02-14 15:23:43,454 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:23:43,454 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:23:43,459 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 15:23:43,463 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:23:43,463 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2470, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 15:23:43,464 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:23:43,464 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2470, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 15:24:21,680 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 15:24:21,680 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 15:24:21,680 - resource_logging.py:150 - __exit__ - DEBUG - Time: 38.21 seconds 2025-02-14 15:24:21,680 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:24:21,680 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30180.06 MB 2025-02-14 15:24:21,680 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38921.25 MB 2025-02-14 15:24:21,680 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8741.19 MB 2025-02-14 15:24:21,680 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 68822.24 MB 2025-02-14 15:24:21,680 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42872.08 MB 2025-02-14 15:24:21,680 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25950.16 MB 2025-02-14 15:24:21,680 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47805.97 MB 2025-02-14 15:24:21,839 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 15:24:21,839 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 15:24:21,839 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 15:24:21,839 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:24:21,839 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38921.25 MB 2025-02-14 15:24:21,839 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28618.61 MB 2025-02-14 15:24:21,839 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -10302.64 MB 2025-02-14 15:24:21,839 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42872.08 MB 2025-02-14 15:24:21,839 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 76183.24 MB 2025-02-14 15:24:21,839 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 33311.16 MB 2025-02-14 15:24:21,839 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 64418.53 MB 2025-02-14 15:24:23,817 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 15:24:23,817 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 15:24:23,817 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.98 seconds 2025-02-14 15:24:23,817 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:24:23,817 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28618.61 MB 2025-02-14 15:24:23,817 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29149.45 MB 2025-02-14 15:24:23,817 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 15:24:23,817 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 76183.24 MB 2025-02-14 15:24:23,817 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31172.07 MB 2025-02-14 15:24:23,817 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -45011.17 MB 2025-02-14 15:24:23,817 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33129.82 MB 2025-02-14 15:24:23,831 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 15:24:23,831 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 15:24:23,831 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:24:23,831 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:24:23,831 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29149.45 MB 2025-02-14 15:24:23,831 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31038.86 MB 2025-02-14 15:24:23,831 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.40 MB 2025-02-14 15:24:23,831 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31172.07 MB 2025-02-14 15:24:23,831 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34475.08 MB 2025-02-14 15:24:23,831 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3303.01 MB 2025-02-14 15:24:23,831 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32456.28 MB 2025-02-14 15:24:24,042 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 15:24:24,042 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 15:24:24,042 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 15:24:24,042 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:24:24,042 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31038.86 MB 2025-02-14 15:24:24,042 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33280.71 MB 2025-02-14 15:24:24,042 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 15:24:24,042 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34475.08 MB 2025-02-14 15:24:24,042 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41081.11 MB 2025-02-14 15:24:24,042 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 15:24:24,042 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38824.99 MB 2025-02-14 15:24:24,043 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 15:24:24,043 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 15:24:24,043 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 15:24:24,043 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:24:24,043 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29149.45 MB 2025-02-14 15:24:24,043 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33280.71 MB 2025-02-14 15:24:24,043 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.26 MB 2025-02-14 15:24:24,043 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31172.07 MB 2025-02-14 15:24:24,043 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41081.11 MB 2025-02-14 15:24:24,043 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9909.04 MB 2025-02-14 15:24:24,043 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38824.99 MB 2025-02-14 15:24:24,212 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 15:24:24,212 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 15:24:24,212 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 15:24:24,212 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:24:24,212 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34814.25 MB 2025-02-14 15:24:24,212 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35581.26 MB 2025-02-14 15:24:24,212 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 15:24:24,212 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41081.11 MB 2025-02-14 15:24:24,212 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41494.25 MB 2025-02-14 15:24:24,212 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 15:24:24,212 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36289.04 MB 2025-02-14 15:24:24,231 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 15:24:24,231 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 15:24:24,231 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:24:24,231 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:24:24,231 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35994.14 MB 2025-02-14 15:24:24,231 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36222.17 MB 2025-02-14 15:24:24,231 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.03 MB 2025-02-14 15:24:24,231 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41494.25 MB 2025-02-14 15:24:24,231 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41494.25 MB 2025-02-14 15:24:24,231 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:24:24,231 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36421.85 MB 2025-02-14 15:24:24,232 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 15:24:24,232 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 15:24:24,232 - resource_logging.py:150 - __exit__ - DEBUG - Time: 40.77 seconds 2025-02-14 15:24:24,232 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:24:24,232 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21574.38 MB 2025-02-14 15:24:24,232 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36422.11 MB 2025-02-14 15:24:24,232 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14847.73 MB 2025-02-14 15:24:24,232 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60213.43 MB 2025-02-14 15:24:24,232 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41494.25 MB 2025-02-14 15:24:24,232 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18719.18 MB 2025-02-14 15:24:24,232 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36422.11 MB 2025-02-14 15:24:24,502 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 15:24:24,502 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 15:24:24,502 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 15:24:24,502 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:24:24,502 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36422.11 MB 2025-02-14 15:24:24,502 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26561.25 MB 2025-02-14 15:24:24,502 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9860.86 MB 2025-02-14 15:24:24,502 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41494.25 MB 2025-02-14 15:24:24,502 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41494.25 MB 2025-02-14 15:24:24,502 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:24:24,502 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38919.65 MB 2025-02-14 15:24:24,520 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8116, cut from 8118 2025-02-14 15:24:24,520 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 15:24:24,526 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 15:24:24,526 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 15:24:24,526 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:24:24,526 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:24:24,526 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26561.25 MB 2025-02-14 15:24:24,526 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34952.81 MB 2025-02-14 15:24:24,526 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8391.56 MB 2025-02-14 15:24:24,526 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41494.25 MB 2025-02-14 15:24:24,526 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45665.48 MB 2025-02-14 15:24:24,526 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4171.24 MB 2025-02-14 15:24:24,526 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34952.81 MB 2025-02-14 15:24:24,687 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7908] 2025-02-14 15:24:24,689 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:24:24,689 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:24:24,690 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:24:24,690 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 15:24:24,694 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 15:24:24,695 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:24:24,695 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 15:24:24,695 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 15:24:26,112 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:24:26,112 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:24:26,117 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 15:24:26,120 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:24:26,120 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 696, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 15:24:26,121 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:24:26,121 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 696, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 15:24:37,037 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 15:24:37,037 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 15:24:37,037 - resource_logging.py:150 - __exit__ - DEBUG - Time: 10.91 seconds 2025-02-14 15:24:37,037 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:24:37,037 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31983.33 MB 2025-02-14 15:24:37,037 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20282.63 MB 2025-02-14 15:24:37,037 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -11700.69 MB 2025-02-14 15:24:37,037 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54007.96 MB 2025-02-14 15:24:37,037 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24962.40 MB 2025-02-14 15:24:37,037 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29045.56 MB 2025-02-14 15:24:37,037 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43123.92 MB 2025-02-14 15:24:37,079 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 15:24:37,079 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 15:24:37,079 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-14 15:24:37,080 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:24:37,080 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20282.63 MB 2025-02-14 15:24:37,080 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19397.19 MB 2025-02-14 15:24:37,080 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -885.45 MB 2025-02-14 15:24:37,080 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24962.40 MB 2025-02-14 15:24:37,080 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31761.37 MB 2025-02-14 15:24:37,080 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6798.97 MB 2025-02-14 15:24:37,080 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29132.71 MB 2025-02-14 15:24:39,009 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 15:24:39,009 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 15:24:39,009 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 15:24:39,009 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:24:39,009 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19397.19 MB 2025-02-14 15:24:39,009 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19928.03 MB 2025-02-14 15:24:39,009 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 15:24:39,009 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31761.37 MB 2025-02-14 15:24:39,009 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24622.66 MB 2025-02-14 15:24:39,009 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7138.71 MB 2025-02-14 15:24:39,009 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23907.36 MB 2025-02-14 15:24:39,029 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 15:24:39,029 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 15:24:39,029 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:24:39,029 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:24:39,029 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19928.03 MB 2025-02-14 15:24:39,029 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21817.56 MB 2025-02-14 15:24:39,029 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 15:24:39,029 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24622.66 MB 2025-02-14 15:24:39,029 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26510.10 MB 2025-02-14 15:24:39,029 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 15:24:39,029 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23234.99 MB 2025-02-14 15:24:39,238 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 15:24:39,238 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 15:24:39,238 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 15:24:39,238 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:24:39,238 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21817.56 MB 2025-02-14 15:24:39,238 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24059.42 MB 2025-02-14 15:24:39,238 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 15:24:39,238 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26510.10 MB 2025-02-14 15:24:39,238 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32172.41 MB 2025-02-14 15:24:39,238 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 15:24:39,238 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29603.70 MB 2025-02-14 15:24:39,239 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 15:24:39,239 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 15:24:39,239 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 15:24:39,239 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:24:39,239 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19928.03 MB 2025-02-14 15:24:39,239 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24059.42 MB 2025-02-14 15:24:39,239 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 15:24:39,239 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24622.66 MB 2025-02-14 15:24:39,239 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32172.41 MB 2025-02-14 15:24:39,239 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-14 15:24:39,239 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29603.70 MB 2025-02-14 15:24:39,404 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 15:24:39,404 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 15:24:39,404 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 15:24:39,404 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:24:39,404 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25592.96 MB 2025-02-14 15:24:39,404 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26359.96 MB 2025-02-14 15:24:39,404 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 15:24:39,404 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32172.41 MB 2025-02-14 15:24:39,404 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32585.55 MB 2025-02-14 15:24:39,404 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 15:24:39,404 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27067.75 MB 2025-02-14 15:24:39,423 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 15:24:39,423 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 15:24:39,423 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:24:39,423 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:24:39,423 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26772.85 MB 2025-02-14 15:24:39,423 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27001.20 MB 2025-02-14 15:24:39,423 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.35 MB 2025-02-14 15:24:39,423 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32585.55 MB 2025-02-14 15:24:39,423 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32585.55 MB 2025-02-14 15:24:39,423 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:24:39,423 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27178.74 MB 2025-02-14 15:24:39,424 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 15:24:39,424 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 15:24:39,424 - resource_logging.py:150 - __exit__ - DEBUG - Time: 13.30 seconds 2025-02-14 15:24:39,424 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:24:39,424 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29558.41 MB 2025-02-14 15:24:39,424 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27201.46 MB 2025-02-14 15:24:39,424 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2356.95 MB 2025-02-14 15:24:39,424 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54007.96 MB 2025-02-14 15:24:39,424 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32585.55 MB 2025-02-14 15:24:39,424 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21422.41 MB 2025-02-14 15:24:39,424 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27201.46 MB 2025-02-14 15:24:39,691 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 15:24:39,691 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 15:24:39,691 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 15:24:39,691 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:24:39,692 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27201.46 MB 2025-02-14 15:24:39,692 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20385.44 MB 2025-02-14 15:24:39,692 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6816.01 MB 2025-02-14 15:24:39,692 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32585.55 MB 2025-02-14 15:24:39,692 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32585.55 MB 2025-02-14 15:24:39,692 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:24:39,692 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29702.99 MB 2025-02-14 15:24:39,710 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8129, cut from 8131 2025-02-14 15:24:39,710 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 15:24:39,716 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 15:24:39,716 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 15:24:39,716 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:24:39,716 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:24:39,716 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20385.44 MB 2025-02-14 15:24:39,716 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28790.56 MB 2025-02-14 15:24:39,716 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8405.11 MB 2025-02-14 15:24:39,716 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32585.55 MB 2025-02-14 15:24:39,716 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40942.70 MB 2025-02-14 15:24:39,716 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8357.15 MB 2025-02-14 15:24:39,716 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28790.56 MB 2025-02-14 15:24:39,877 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7921] 2025-02-14 15:24:39,878 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:24:39,878 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:24:39,879 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:24:39,879 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 15:24:39,884 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 15:24:39,885 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:24:39,885 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 15:24:39,885 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 15:24:57,954 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:24:57,954 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:24:57,959 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 15:24:57,960 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:24:57,960 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 158, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 15:24:57,961 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:24:57,961 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 158, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 15:25:00,469 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 15:25:00,469 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 15:25:00,469 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.50 seconds 2025-02-14 15:25:00,469 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:25:00,469 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16494.80 MB 2025-02-14 15:25:00,469 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17053.96 MB 2025-02-14 15:25:00,469 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 559.15 MB 2025-02-14 15:25:00,469 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53477.38 MB 2025-02-14 15:25:00,469 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19268.63 MB 2025-02-14 15:25:00,469 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34208.74 MB 2025-02-14 15:25:00,469 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25966.98 MB 2025-02-14 15:25:00,482 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 15:25:00,482 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 15:25:00,482 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:25:00,482 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:25:00,482 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17053.96 MB 2025-02-14 15:25:00,482 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17325.59 MB 2025-02-14 15:25:00,482 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 271.63 MB 2025-02-14 15:25:00,482 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19268.63 MB 2025-02-14 15:25:00,482 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20942.16 MB 2025-02-14 15:25:00,482 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1673.53 MB 2025-02-14 15:25:00,482 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19324.35 MB 2025-02-14 15:25:01,251 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 15:25:01,251 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 15:25:01,251 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.77 seconds 2025-02-14 15:25:01,251 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:25:01,251 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17325.59 MB 2025-02-14 15:25:01,251 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17535.27 MB 2025-02-14 15:25:01,251 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 209.68 MB 2025-02-14 15:25:01,251 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20942.16 MB 2025-02-14 15:25:01,251 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19910.36 MB 2025-02-14 15:25:01,251 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1031.80 MB 2025-02-14 15:25:01,251 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21497.06 MB 2025-02-14 15:25:01,259 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 15:25:01,259 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 15:25:01,259 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 15:25:01,259 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:25:01,259 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17535.27 MB 2025-02-14 15:25:01,259 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18281.45 MB 2025-02-14 15:25:01,259 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 746.18 MB 2025-02-14 15:25:01,259 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19910.36 MB 2025-02-14 15:25:01,259 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19910.36 MB 2025-02-14 15:25:01,259 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:25:01,259 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 18841.34 MB 2025-02-14 15:25:01,347 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 15:25:01,348 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 15:25:01,348 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 15:25:01,348 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:25:01,348 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18281.45 MB 2025-02-14 15:25:01,348 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19167.55 MB 2025-02-14 15:25:01,348 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 886.10 MB 2025-02-14 15:25:01,348 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19910.36 MB 2025-02-14 15:25:01,348 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22523.41 MB 2025-02-14 15:25:01,348 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2613.05 MB 2025-02-14 15:25:01,348 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21360.12 MB 2025-02-14 15:25:01,348 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 15:25:01,348 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 15:25:01,348 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 15:25:01,348 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:25:01,348 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17535.27 MB 2025-02-14 15:25:01,348 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19167.55 MB 2025-02-14 15:25:01,348 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1632.28 MB 2025-02-14 15:25:01,348 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19910.36 MB 2025-02-14 15:25:01,348 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22523.41 MB 2025-02-14 15:25:01,348 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2613.05 MB 2025-02-14 15:25:01,348 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21360.12 MB 2025-02-14 15:25:01,414 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 15:25:01,414 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 15:25:01,414 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 15:25:01,415 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:25:01,415 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19447.13 MB 2025-02-14 15:25:01,415 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19750.35 MB 2025-02-14 15:25:01,415 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 303.23 MB 2025-02-14 15:25:01,415 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22523.41 MB 2025-02-14 15:25:01,415 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22684.89 MB 2025-02-14 15:25:01,415 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 161.48 MB 2025-02-14 15:25:01,415 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20039.12 MB 2025-02-14 15:25:01,423 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 15:25:01,423 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 15:25:01,423 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:25:01,423 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:25:01,423 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19913.45 MB 2025-02-14 15:25:01,423 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20114.38 MB 2025-02-14 15:25:01,423 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 200.92 MB 2025-02-14 15:25:01,423 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22684.89 MB 2025-02-14 15:25:01,423 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22684.89 MB 2025-02-14 15:25:01,423 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:25:01,423 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20134.29 MB 2025-02-14 15:25:01,424 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 15:25:01,424 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 15:25:01,424 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.46 seconds 2025-02-14 15:25:01,424 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:25:01,424 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 15944.32 MB 2025-02-14 15:25:01,424 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20315.45 MB 2025-02-14 15:25:01,424 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4371.13 MB 2025-02-14 15:25:01,424 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53477.38 MB 2025-02-14 15:25:01,424 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22686.99 MB 2025-02-14 15:25:01,424 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30790.39 MB 2025-02-14 15:25:01,424 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20315.45 MB 2025-02-14 15:25:01,689 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 15:25:01,689 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 15:25:01,689 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 15:25:01,689 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:25:01,689 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20315.45 MB 2025-02-14 15:25:01,689 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20415.92 MB 2025-02-14 15:25:01,689 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.47 MB 2025-02-14 15:25:01,689 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22686.99 MB 2025-02-14 15:25:01,689 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22686.99 MB 2025-02-14 15:25:01,689 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:25:01,689 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21018.72 MB 2025-02-14 15:25:01,707 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 15:25:01,708 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2.'] 2025-02-14 15:25:01,714 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 15:25:01,714 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 15:25:01,714 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:25:01,714 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:25:01,714 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16565.04 MB 2025-02-14 15:25:01,714 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20759.53 MB 2025-02-14 15:25:01,714 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4194.49 MB 2025-02-14 15:25:01,714 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22686.99 MB 2025-02-14 15:25:01,714 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33176.94 MB 2025-02-14 15:25:01,714 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 15:25:01,714 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24953.83 MB 2025-02-14 15:25:01,877 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 15:25:01,878 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:25:01,878 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:25:01,879 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:25:01,879 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 15:25:01,884 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 15:25:01,885 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:25:01,885 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 15:25:01,885 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2.'] 2025-02-14 15:25:01,886 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:25:01,886 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:25:01,886 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:25:01,886 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:25:01,892 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 15:25:01,893 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:25:01,893 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:25:01,893 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:25:01,893 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:25:01,893 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 15:25:01,894 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:25:01,894 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:25:01,894 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:25:01,894 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 15:25:01,894 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 15:25:01,895 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:25:01,895 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:25:01,900 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:25:01,900 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:25:01,901 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:25:01,901 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:25:01,902 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:25:01,902 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:25:01,904 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:25:01,904 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:25:12,658 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:25:12,658 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:25:12,666 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 15:25:12,668 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:25:12,668 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 177, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 15:25:12,670 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:25:12,670 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 177, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 15:25:15,481 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 15:25:15,481 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 15:25:15,481 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.80 seconds 2025-02-14 15:25:15,481 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:25:15,481 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16748.92 MB 2025-02-14 15:25:15,481 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17375.32 MB 2025-02-14 15:25:15,481 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 626.39 MB 2025-02-14 15:25:15,481 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33176.94 MB 2025-02-14 15:25:15,481 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19818.09 MB 2025-02-14 15:25:15,481 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13358.86 MB 2025-02-14 15:25:15,481 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26221.10 MB 2025-02-14 15:25:15,501 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 15:25:15,501 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 15:25:15,501 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:25:15,501 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:25:15,501 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17375.32 MB 2025-02-14 15:25:15,501 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17679.99 MB 2025-02-14 15:25:15,501 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 304.68 MB 2025-02-14 15:25:15,501 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19818.09 MB 2025-02-14 15:25:15,501 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21376.27 MB 2025-02-14 15:25:15,501 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1558.18 MB 2025-02-14 15:25:15,501 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19867.73 MB 2025-02-14 15:25:16,366 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 15:25:16,367 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 15:25:16,367 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.86 seconds 2025-02-14 15:25:16,367 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:25:16,367 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17679.99 MB 2025-02-14 15:25:16,367 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17914.89 MB 2025-02-14 15:25:16,367 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 234.90 MB 2025-02-14 15:25:16,367 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21376.27 MB 2025-02-14 15:25:16,367 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20375.93 MB 2025-02-14 15:25:16,367 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1000.34 MB 2025-02-14 15:25:16,367 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21851.47 MB 2025-02-14 15:25:16,375 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 15:25:16,375 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 15:25:16,375 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:25:16,375 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:25:16,375 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17914.89 MB 2025-02-14 15:25:16,375 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18750.81 MB 2025-02-14 15:25:16,375 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 835.92 MB 2025-02-14 15:25:16,375 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20375.93 MB 2025-02-14 15:25:16,375 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20795.36 MB 2025-02-14 15:25:16,375 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 419.43 MB 2025-02-14 15:25:16,375 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19378.02 MB 2025-02-14 15:25:16,473 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 15:25:16,473 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 15:25:16,473 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 15:25:16,473 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:25:16,473 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18750.81 MB 2025-02-14 15:25:16,473 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19742.87 MB 2025-02-14 15:25:16,473 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 992.06 MB 2025-02-14 15:25:16,473 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20795.36 MB 2025-02-14 15:25:16,473 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23521.66 MB 2025-02-14 15:25:16,473 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2726.30 MB 2025-02-14 15:25:16,473 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22197.09 MB 2025-02-14 15:25:16,474 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 15:25:16,474 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 15:25:16,474 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 15:25:16,474 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:25:16,474 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17914.89 MB 2025-02-14 15:25:16,474 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19742.87 MB 2025-02-14 15:25:16,474 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1827.97 MB 2025-02-14 15:25:16,474 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20375.93 MB 2025-02-14 15:25:16,474 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23521.66 MB 2025-02-14 15:25:16,474 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3145.73 MB 2025-02-14 15:25:16,474 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22197.09 MB 2025-02-14 15:25:16,547 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 15:25:16,547 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 15:25:16,547 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 15:25:16,547 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:25:16,547 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20056.06 MB 2025-02-14 15:25:16,547 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20396.38 MB 2025-02-14 15:25:16,547 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 340.32 MB 2025-02-14 15:25:16,547 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23521.66 MB 2025-02-14 15:25:16,547 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23702.01 MB 2025-02-14 15:25:16,547 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 180.36 MB 2025-02-14 15:25:16,547 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20716.20 MB 2025-02-14 15:25:16,556 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 15:25:16,556 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 15:25:16,556 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:25:16,556 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:25:16,556 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20579.09 MB 2025-02-14 15:25:16,556 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20785.79 MB 2025-02-14 15:25:16,556 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.70 MB 2025-02-14 15:25:16,556 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23702.01 MB 2025-02-14 15:25:16,556 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23704.11 MB 2025-02-14 15:25:16,557 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-14 15:25:16,557 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20814.61 MB 2025-02-14 15:25:16,558 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 15:25:16,558 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 15:25:16,558 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.88 seconds 2025-02-14 15:25:16,558 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:25:16,558 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16132.24 MB 2025-02-14 15:25:16,558 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20986.86 MB 2025-02-14 15:25:16,558 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4854.62 MB 2025-02-14 15:25:16,558 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33176.94 MB 2025-02-14 15:25:16,558 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23704.11 MB 2025-02-14 15:25:16,558 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9472.84 MB 2025-02-14 15:25:16,558 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20986.86 MB 2025-02-14 15:25:16,822 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 15:25:16,823 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 15:25:16,823 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 15:25:16,823 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:25:16,823 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20986.86 MB 2025-02-14 15:25:16,823 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21087.33 MB 2025-02-14 15:25:16,823 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.47 MB 2025-02-14 15:25:16,823 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23704.11 MB 2025-02-14 15:25:16,823 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23704.11 MB 2025-02-14 15:25:16,823 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:25:16,823 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21690.13 MB 2025-02-14 15:25:16,841 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 15:25:16,841 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2.'] 2025-02-14 15:25:16,847 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 15:25:16,847 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 15:25:16,847 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:25:16,847 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:25:16,847 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16804.06 MB 2025-02-14 15:25:16,847 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20998.55 MB 2025-02-14 15:25:16,847 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4194.49 MB 2025-02-14 15:25:16,847 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23704.11 MB 2025-02-14 15:25:16,847 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34194.06 MB 2025-02-14 15:25:16,847 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 15:25:16,847 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25192.85 MB 2025-02-14 15:25:17,009 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 15:25:17,011 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:25:17,011 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:25:17,011 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:25:17,011 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 15:25:17,016 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 15:25:17,017 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:25:17,017 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 15:25:17,017 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2.'] 2025-02-14 15:25:17,018 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:25:17,018 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:25:17,019 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:25:17,019 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:25:17,024 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 15:25:17,025 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:25:17,025 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:25:17,025 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:25:17,025 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:25:17,025 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 15:25:17,026 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:25:17,026 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:25:17,026 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:25:17,026 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 15:25:17,026 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 15:25:17,027 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:25:17,027 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:25:17,032 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:25:17,032 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:25:17,033 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:25:17,033 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:25:17,034 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:25:17,034 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:25:17,037 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:25:17,037 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:25:25,338 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:25:25,339 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:25:25,344 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 15:25:25,345 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:25:25,345 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 208, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 15:25:25,347 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:25:25,347 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 208, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 15:25:28,615 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 15:25:28,615 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 15:25:28,615 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.26 seconds 2025-02-14 15:25:28,615 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:25:28,615 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17208.39 MB 2025-02-14 15:25:28,615 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17944.49 MB 2025-02-14 15:25:28,615 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 736.10 MB 2025-02-14 15:25:28,615 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34194.06 MB 2025-02-14 15:25:28,615 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20484.98 MB 2025-02-14 15:25:28,615 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13709.08 MB 2025-02-14 15:25:28,615 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26907.06 MB 2025-02-14 15:25:28,632 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 15:25:28,632 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 15:25:28,632 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:25:28,632 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:25:28,632 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17944.49 MB 2025-02-14 15:25:28,632 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18301.06 MB 2025-02-14 15:25:28,632 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 356.57 MB 2025-02-14 15:25:28,632 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20484.98 MB 2025-02-14 15:25:28,632 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21957.18 MB 2025-02-14 15:25:28,632 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1472.20 MB 2025-02-14 15:25:28,632 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20923.48 MB 2025-02-14 15:25:29,640 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 15:25:29,640 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 15:25:29,640 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.01 seconds 2025-02-14 15:25:29,640 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:25:29,640 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18301.06 MB 2025-02-14 15:25:29,640 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18577.10 MB 2025-02-14 15:25:29,640 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 276.04 MB 2025-02-14 15:25:29,640 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21957.18 MB 2025-02-14 15:25:29,640 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20111.69 MB 2025-02-14 15:25:29,640 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1845.49 MB 2025-02-14 15:25:29,640 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22558.51 MB 2025-02-14 15:25:29,649 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 15:25:29,649 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 15:25:29,649 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:25:29,649 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:25:29,650 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18577.10 MB 2025-02-14 15:25:29,650 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19559.42 MB 2025-02-14 15:25:29,650 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 982.32 MB 2025-02-14 15:25:29,650 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20111.69 MB 2025-02-14 15:25:29,650 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21583.89 MB 2025-02-14 15:25:29,650 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1472.20 MB 2025-02-14 15:25:29,650 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20296.49 MB 2025-02-14 15:25:29,774 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 15:25:29,774 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 15:25:29,774 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 15:25:29,774 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:25:29,774 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19559.42 MB 2025-02-14 15:25:29,774 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20725.74 MB 2025-02-14 15:25:29,774 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1166.32 MB 2025-02-14 15:25:29,774 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21583.89 MB 2025-02-14 15:25:29,774 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24773.66 MB 2025-02-14 15:25:29,774 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3189.77 MB 2025-02-14 15:25:29,774 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23608.74 MB 2025-02-14 15:25:29,775 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 15:25:29,775 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 15:25:29,775 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 15:25:29,775 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:25:29,775 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18577.10 MB 2025-02-14 15:25:29,775 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20725.74 MB 2025-02-14 15:25:29,775 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2148.64 MB 2025-02-14 15:25:29,775 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20111.69 MB 2025-02-14 15:25:29,775 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24773.66 MB 2025-02-14 15:25:29,775 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4661.97 MB 2025-02-14 15:25:29,775 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23608.74 MB 2025-02-14 15:25:29,860 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 15:25:29,860 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 15:25:29,860 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 15:25:29,860 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:25:29,860 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21093.79 MB 2025-02-14 15:25:29,860 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21492.63 MB 2025-02-14 15:25:29,860 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 398.84 MB 2025-02-14 15:25:29,860 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24773.66 MB 2025-02-14 15:25:29,860 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24987.57 MB 2025-02-14 15:25:29,860 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 213.91 MB 2025-02-14 15:25:29,860 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21862.33 MB 2025-02-14 15:25:29,871 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 15:25:29,871 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 15:25:29,871 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:25:29,871 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:25:29,871 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21707.34 MB 2025-02-14 15:25:29,871 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21914.13 MB 2025-02-14 15:25:29,871 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.79 MB 2025-02-14 15:25:29,871 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24987.57 MB 2025-02-14 15:25:29,871 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24987.57 MB 2025-02-14 15:25:29,871 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:25:29,871 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21966.46 MB 2025-02-14 15:25:29,872 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 15:25:29,872 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 15:25:29,872 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.52 seconds 2025-02-14 15:25:29,872 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:25:29,872 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16483.70 MB 2025-02-14 15:25:29,872 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22114.81 MB 2025-02-14 15:25:29,872 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5631.11 MB 2025-02-14 15:25:29,872 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34194.06 MB 2025-02-14 15:25:29,872 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24987.57 MB 2025-02-14 15:25:29,872 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9206.50 MB 2025-02-14 15:25:29,872 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22114.81 MB 2025-02-14 15:25:30,137 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 15:25:30,137 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 15:25:30,137 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 15:25:30,137 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:25:30,137 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17136.24 MB 2025-02-14 15:25:30,137 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17236.51 MB 2025-02-14 15:25:30,137 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.27 MB 2025-02-14 15:25:30,137 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24987.57 MB 2025-02-14 15:25:30,137 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24987.57 MB 2025-02-14 15:25:30,137 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:25:30,137 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 17838.13 MB 2025-02-14 15:25:30,155 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8146, cut from 8148 2025-02-14 15:25:30,155 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-14 15:25:30,161 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 15:25:30,161 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 15:25:30,161 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:25:30,161 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:25:30,161 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17236.51 MB 2025-02-14 15:25:30,161 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21422.79 MB 2025-02-14 15:25:30,161 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4186.28 MB 2025-02-14 15:25:30,161 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24987.57 MB 2025-02-14 15:25:30,161 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35456.55 MB 2025-02-14 15:25:30,161 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10468.98 MB 2025-02-14 15:25:30,162 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25608.70 MB 2025-02-14 15:25:30,323 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7938] 2025-02-14 15:25:30,324 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:25:30,324 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:25:30,325 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:25:30,325 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 15:25:30,330 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 15:25:30,331 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:25:30,331 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 15:25:30,331 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-14 15:25:30,332 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:25:30,332 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:25:30,332 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:25:30,332 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:25:30,338 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 15:25:30,339 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:25:30,339 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:25:30,339 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:25:30,339 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:25:30,339 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 15:25:30,340 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:25:30,340 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:25:30,340 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:25:30,340 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 15:25:30,340 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 15:25:30,341 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:25:30,341 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:25:30,346 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:25:30,346 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:25:30,347 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:25:30,347 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:25:30,348 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:25:30,348 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:25:30,349 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:25:30,349 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:25:51,293 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:25:51,293 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:25:51,298 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 15:25:51,299 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:25:51,299 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 161, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 15:25:51,300 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:25:51,300 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 161, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 15:25:53,809 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 15:25:53,809 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 15:25:53,809 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.51 seconds 2025-02-14 15:25:53,809 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:25:53,809 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17002.61 MB 2025-02-14 15:25:53,809 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17573.04 MB 2025-02-14 15:25:53,809 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 570.43 MB 2025-02-14 15:25:53,809 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35456.55 MB 2025-02-14 15:25:53,809 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20436.75 MB 2025-02-14 15:25:53,809 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15019.80 MB 2025-02-14 15:25:53,809 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26474.79 MB 2025-02-14 15:25:53,822 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 15:25:53,822 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 15:25:53,822 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:25:53,822 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:25:53,822 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17573.04 MB 2025-02-14 15:25:53,822 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17814.28 MB 2025-02-14 15:25:53,822 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 241.24 MB 2025-02-14 15:25:53,822 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20436.75 MB 2025-02-14 15:25:53,822 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21258.83 MB 2025-02-14 15:25:53,822 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 822.08 MB 2025-02-14 15:25:53,822 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19768.85 MB 2025-02-14 15:25:54,603 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 15:25:54,604 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 15:25:54,604 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.78 seconds 2025-02-14 15:25:54,604 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:25:54,604 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17814.28 MB 2025-02-14 15:25:54,604 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18021.30 MB 2025-02-14 15:25:54,604 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 207.03 MB 2025-02-14 15:25:54,604 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21258.83 MB 2025-02-14 15:25:54,604 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20315.11 MB 2025-02-14 15:25:54,604 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -943.72 MB 2025-02-14 15:25:54,604 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21985.75 MB 2025-02-14 15:25:54,612 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 15:25:54,612 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 15:25:54,612 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 15:25:54,612 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:25:54,612 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18021.24 MB 2025-02-14 15:25:54,612 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18759.03 MB 2025-02-14 15:25:54,612 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 737.79 MB 2025-02-14 15:25:54,612 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20315.11 MB 2025-02-14 15:25:54,612 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20684.21 MB 2025-02-14 15:25:54,612 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 369.10 MB 2025-02-14 15:25:54,612 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19311.83 MB 2025-02-14 15:25:54,695 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 15:25:54,695 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 15:25:54,695 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 15:25:54,695 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:25:54,695 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18759.03 MB 2025-02-14 15:25:54,695 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19633.39 MB 2025-02-14 15:25:54,695 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 874.36 MB 2025-02-14 15:25:54,695 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20684.21 MB 2025-02-14 15:25:54,696 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23083.35 MB 2025-02-14 15:25:54,696 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2399.14 MB 2025-02-14 15:25:54,696 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21801.39 MB 2025-02-14 15:25:54,696 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 15:25:54,696 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 15:25:54,696 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 15:25:54,696 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:25:54,696 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18021.24 MB 2025-02-14 15:25:54,696 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19633.39 MB 2025-02-14 15:25:54,696 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1612.15 MB 2025-02-14 15:25:54,696 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20315.11 MB 2025-02-14 15:25:54,696 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23083.35 MB 2025-02-14 15:25:54,696 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2768.24 MB 2025-02-14 15:25:54,696 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21801.39 MB 2025-02-14 15:25:54,758 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 15:25:54,758 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 15:25:54,758 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 15:25:54,758 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:25:54,758 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19909.43 MB 2025-02-14 15:25:54,758 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20209.61 MB 2025-02-14 15:25:54,758 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 300.18 MB 2025-02-14 15:25:54,758 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23083.35 MB 2025-02-14 15:25:54,758 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23242.74 MB 2025-02-14 15:25:54,758 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 159.38 MB 2025-02-14 15:25:54,758 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20493.80 MB 2025-02-14 15:25:54,767 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 15:25:54,767 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 15:25:54,767 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:25:54,767 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:25:54,767 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20370.64 MB 2025-02-14 15:25:54,767 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20571.72 MB 2025-02-14 15:25:54,767 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 201.08 MB 2025-02-14 15:25:54,767 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23242.74 MB 2025-02-14 15:25:54,767 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23242.74 MB 2025-02-14 15:25:54,767 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:25:54,767 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20584.02 MB 2025-02-14 15:25:54,768 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 15:25:54,768 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 15:25:54,768 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.47 seconds 2025-02-14 15:25:54,768 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:25:54,768 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16441.67 MB 2025-02-14 15:25:54,768 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20772.74 MB 2025-02-14 15:25:54,768 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4331.06 MB 2025-02-14 15:25:54,768 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35456.55 MB 2025-02-14 15:25:54,768 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23242.74 MB 2025-02-14 15:25:54,768 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12213.81 MB 2025-02-14 15:25:54,768 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20772.74 MB 2025-02-14 15:25:55,030 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 15:25:55,030 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 15:25:55,030 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 15:25:55,030 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:25:55,030 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20772.74 MB 2025-02-14 15:25:55,031 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20873.18 MB 2025-02-14 15:25:55,031 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.44 MB 2025-02-14 15:25:55,031 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23242.74 MB 2025-02-14 15:25:55,031 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23242.74 MB 2025-02-14 15:25:55,031 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:25:55,031 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21476.64 MB 2025-02-14 15:25:55,049 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8160, cut from 8162 2025-02-14 15:25:55,049 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for the video is 2.'] 2025-02-14 15:25:55,055 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 15:25:55,055 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 15:25:55,055 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:25:55,055 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:25:55,055 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17057.83 MB 2025-02-14 15:25:55,055 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21252.13 MB 2025-02-14 15:25:55,055 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4194.31 MB 2025-02-14 15:25:55,055 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23242.74 MB 2025-02-14 15:25:55,055 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33728.50 MB 2025-02-14 15:25:55,055 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10485.76 MB 2025-02-14 15:25:55,055 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25445.08 MB 2025-02-14 15:25:55,205 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7952] 2025-02-14 15:25:55,206 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:25:55,206 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:25:55,207 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:25:55,207 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 15:25:55,211 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 15:25:55,212 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:25:55,212 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 15:25:55,212 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for the video is 2.'] 2025-02-14 15:25:55,213 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:25:55,213 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:25:55,214 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:25:55,214 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:25:55,219 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 15:25:55,220 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:25:55,220 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:25:55,220 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:25:55,220 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:25:55,220 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 15:25:55,221 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:25:55,221 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:25:55,221 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:25:55,221 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 15:25:55,221 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 15:25:55,222 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:25:55,222 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:25:55,225 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:25:55,225 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:25:55,226 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:25:55,226 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:25:55,227 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:25:55,227 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:25:55,228 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:25:55,228 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:26:05,197 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:05,197 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:26:05,202 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 15:26:05,203 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:05,203 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 472, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 15:26:05,204 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:05,204 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 472, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 15:26:12,607 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 15:26:12,608 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 15:26:12,608 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.40 seconds 2025-02-14 15:26:12,608 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:26:12,608 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24784.71 MB 2025-02-14 15:26:12,608 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26455.09 MB 2025-02-14 15:26:12,608 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1670.38 MB 2025-02-14 15:26:12,608 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33728.50 MB 2025-02-14 15:26:12,608 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31081.89 MB 2025-02-14 15:26:12,608 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2646.61 MB 2025-02-14 15:26:12,608 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35388.54 MB 2025-02-14 15:26:12,643 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 15:26:12,643 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 15:26:12,643 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 15:26:12,643 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:26:12,643 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26455.09 MB 2025-02-14 15:26:12,643 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26759.71 MB 2025-02-14 15:26:12,643 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 304.62 MB 2025-02-14 15:26:12,643 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31081.89 MB 2025-02-14 15:26:12,643 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37025.22 MB 2025-02-14 15:26:12,643 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5943.33 MB 2025-02-14 15:26:12,643 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33919.52 MB 2025-02-14 15:26:14,554 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 15:26:14,554 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 15:26:14,554 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 15:26:14,554 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:26:14,554 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26759.71 MB 2025-02-14 15:26:14,554 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27290.56 MB 2025-02-14 15:26:14,554 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 15:26:14,554 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37025.22 MB 2025-02-14 15:26:14,554 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33206.30 MB 2025-02-14 15:26:14,554 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3818.91 MB 2025-02-14 15:26:14,554 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31269.89 MB 2025-02-14 15:26:14,568 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 15:26:14,568 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 15:26:14,568 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:26:14,568 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:26:14,568 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27290.56 MB 2025-02-14 15:26:14,568 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29180.09 MB 2025-02-14 15:26:14,568 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 15:26:14,568 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33206.30 MB 2025-02-14 15:26:14,568 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34150.02 MB 2025-02-14 15:26:14,568 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 15:26:14,568 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30597.52 MB 2025-02-14 15:26:14,780 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 15:26:14,780 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 15:26:14,780 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 15:26:14,780 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:26:14,780 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29180.09 MB 2025-02-14 15:26:14,780 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25928.67 MB 2025-02-14 15:26:14,780 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3251.42 MB 2025-02-14 15:26:14,780 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34150.02 MB 2025-02-14 15:26:14,780 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36037.46 MB 2025-02-14 15:26:14,780 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 15:26:14,780 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31472.95 MB 2025-02-14 15:26:14,781 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 15:26:14,781 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 15:26:14,781 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 15:26:14,781 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:26:14,781 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27290.56 MB 2025-02-14 15:26:14,781 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25928.67 MB 2025-02-14 15:26:14,781 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1361.89 MB 2025-02-14 15:26:14,781 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33206.30 MB 2025-02-14 15:26:14,781 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36037.46 MB 2025-02-14 15:26:14,781 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-14 15:26:14,781 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31472.95 MB 2025-02-14 15:26:14,943 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 15:26:14,943 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 15:26:14,943 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 15:26:14,943 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:26:14,943 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26636.46 MB 2025-02-14 15:26:14,943 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27403.46 MB 2025-02-14 15:26:14,943 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 15:26:14,944 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36037.46 MB 2025-02-14 15:26:14,944 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36452.70 MB 2025-02-14 15:26:14,944 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 15:26:14,944 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28111.25 MB 2025-02-14 15:26:14,961 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 15:26:14,961 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 15:26:14,961 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:26:14,961 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:26:14,961 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27816.35 MB 2025-02-14 15:26:14,961 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28022.42 MB 2025-02-14 15:26:14,961 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.07 MB 2025-02-14 15:26:14,961 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36452.70 MB 2025-02-14 15:26:14,961 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36452.70 MB 2025-02-14 15:26:14,961 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:26:14,961 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28224.26 MB 2025-02-14 15:26:14,962 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 15:26:14,962 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 15:26:14,962 - resource_logging.py:150 - __exit__ - DEBUG - Time: 9.76 seconds 2025-02-14 15:26:14,962 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:26:14,962 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23140.23 MB 2025-02-14 15:26:14,962 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28223.22 MB 2025-02-14 15:26:14,962 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5082.99 MB 2025-02-14 15:26:14,962 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33728.50 MB 2025-02-14 15:26:14,962 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36452.70 MB 2025-02-14 15:26:14,962 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2724.20 MB 2025-02-14 15:26:14,962 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28224.26 MB 2025-02-14 15:26:15,227 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 15:26:15,227 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 15:26:15,227 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 15:26:15,227 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:26:15,227 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28223.22 MB 2025-02-14 15:26:15,227 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28323.55 MB 2025-02-14 15:26:15,227 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.33 MB 2025-02-14 15:26:15,227 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36452.70 MB 2025-02-14 15:26:15,227 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36452.70 MB 2025-02-14 15:26:15,227 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:26:15,227 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28925.54 MB 2025-02-14 15:26:15,244 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8151, cut from 8153 2025-02-14 15:26:15,244 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-14 15:26:15,251 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 15:26:15,251 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 15:26:15,251 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:26:15,251 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:26:15,251 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18909.60 MB 2025-02-14 15:26:15,251 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23098.44 MB 2025-02-14 15:26:15,251 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4188.84 MB 2025-02-14 15:26:15,251 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36452.70 MB 2025-02-14 15:26:15,251 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40642.81 MB 2025-02-14 15:26:15,251 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4190.11 MB 2025-02-14 15:26:15,251 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27286.77 MB 2025-02-14 15:26:15,413 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7943] 2025-02-14 15:26:15,414 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:15,415 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:26:15,415 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:15,415 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 15:26:15,420 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 15:26:15,421 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:15,421 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 15:26:15,421 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-14 15:26:15,422 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:15,422 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:26:15,423 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:15,423 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:26:15,429 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 15:26:15,429 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:15,429 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:26:15,430 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:15,430 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:26:15,430 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 15:26:15,430 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:15,430 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:26:15,431 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:15,431 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 15:26:15,431 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 15:26:15,431 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:15,431 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:26:15,437 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:15,437 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:26:15,438 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:15,438 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:26:15,439 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:15,439 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:26:15,441 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:15,441 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:26:22,682 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:22,682 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:26:22,687 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 15:26:22,688 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:22,688 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 195, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 15:26:22,689 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:22,689 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 195, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 15:26:25,754 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 15:26:25,754 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 15:26:25,754 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.06 seconds 2025-02-14 15:26:25,754 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:26:25,754 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17482.98 MB 2025-02-14 15:26:25,754 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18173.08 MB 2025-02-14 15:26:25,754 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 690.09 MB 2025-02-14 15:26:25,754 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40642.81 MB 2025-02-14 15:26:25,754 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23379.05 MB 2025-02-14 15:26:25,754 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17263.76 MB 2025-02-14 15:26:25,754 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27180.84 MB 2025-02-14 15:26:25,768 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 15:26:25,768 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 15:26:25,768 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:26:25,768 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:26:25,768 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18173.08 MB 2025-02-14 15:26:25,768 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18387.97 MB 2025-02-14 15:26:25,768 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 214.89 MB 2025-02-14 15:26:25,768 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23379.05 MB 2025-02-14 15:26:25,768 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23379.05 MB 2025-02-14 15:26:25,768 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:26:25,768 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20684.17 MB 2025-02-14 15:26:26,620 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 15:26:26,620 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 15:26:26,620 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.85 seconds 2025-02-14 15:26:26,620 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:26:26,620 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18387.97 MB 2025-02-14 15:26:26,620 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18624.19 MB 2025-02-14 15:26:26,620 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 236.22 MB 2025-02-14 15:26:26,620 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23379.05 MB 2025-02-14 15:26:26,620 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23379.05 MB 2025-02-14 15:26:26,620 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:26:26,620 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22558.40 MB 2025-02-14 15:26:26,628 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 15:26:26,628 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 15:26:26,628 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:26:26,628 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:26:26,628 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18624.19 MB 2025-02-14 15:26:26,628 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19464.83 MB 2025-02-14 15:26:26,628 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 840.64 MB 2025-02-14 15:26:26,628 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23379.05 MB 2025-02-14 15:26:26,628 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 23379.05 MB 2025-02-14 15:26:26,628 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:26:26,628 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20095.59 MB 2025-02-14 15:26:26,724 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 15:26:26,724 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 15:26:26,724 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 15:26:26,724 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:26:26,724 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 19464.83 MB 2025-02-14 15:26:26,724 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20462.49 MB 2025-02-14 15:26:26,724 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 997.66 MB 2025-02-14 15:26:26,724 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23379.05 MB 2025-02-14 15:26:26,724 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24643.63 MB 2025-02-14 15:26:26,724 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1264.58 MB 2025-02-14 15:26:26,724 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22929.66 MB 2025-02-14 15:26:26,724 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 15:26:26,724 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 15:26:26,724 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 15:26:26,725 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:26:26,725 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18624.19 MB 2025-02-14 15:26:26,725 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20462.49 MB 2025-02-14 15:26:26,725 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1838.30 MB 2025-02-14 15:26:26,725 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 23379.05 MB 2025-02-14 15:26:26,725 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24643.63 MB 2025-02-14 15:26:26,725 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1264.58 MB 2025-02-14 15:26:26,725 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22929.66 MB 2025-02-14 15:26:26,798 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 15:26:26,798 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 15:26:26,798 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 15:26:26,798 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:26:26,798 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20777.46 MB 2025-02-14 15:26:26,798 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21119.37 MB 2025-02-14 15:26:26,798 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 341.91 MB 2025-02-14 15:26:26,798 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24643.63 MB 2025-02-14 15:26:26,798 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24826.09 MB 2025-02-14 15:26:26,798 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 182.45 MB 2025-02-14 15:26:26,798 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21438.86 MB 2025-02-14 15:26:26,807 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 15:26:26,807 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 15:26:26,807 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:26:26,807 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:26:26,807 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21303.11 MB 2025-02-14 15:26:26,807 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21504.70 MB 2025-02-14 15:26:26,807 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 201.59 MB 2025-02-14 15:26:26,807 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24826.09 MB 2025-02-14 15:26:26,807 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24830.28 MB 2025-02-14 15:26:26,807 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-14 15:26:26,807 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21537.05 MB 2025-02-14 15:26:26,808 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 15:26:26,808 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 15:26:26,809 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.12 seconds 2025-02-14 15:26:26,809 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:26:26,809 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16803.59 MB 2025-02-14 15:26:26,809 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21705.77 MB 2025-02-14 15:26:26,809 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4902.18 MB 2025-02-14 15:26:26,809 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40642.81 MB 2025-02-14 15:26:26,809 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24830.28 MB 2025-02-14 15:26:26,809 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15812.53 MB 2025-02-14 15:26:26,809 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21705.77 MB 2025-02-14 15:26:27,073 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 15:26:27,073 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 15:26:27,073 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 15:26:27,073 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:26:27,073 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21705.77 MB 2025-02-14 15:26:27,073 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21806.24 MB 2025-02-14 15:26:27,073 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.47 MB 2025-02-14 15:26:27,073 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24830.28 MB 2025-02-14 15:26:27,073 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 24830.28 MB 2025-02-14 15:26:27,073 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:26:27,073 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22409.04 MB 2025-02-14 15:26:27,091 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 15:26:27,091 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-14 15:26:27,097 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 15:26:27,097 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 15:26:27,097 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:26:27,097 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:26:27,097 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17477.73 MB 2025-02-14 15:26:27,097 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21672.22 MB 2025-02-14 15:26:27,097 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4194.49 MB 2025-02-14 15:26:27,097 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 24830.28 MB 2025-02-14 15:26:27,097 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33220.98 MB 2025-02-14 15:26:27,097 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 15:26:27,097 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25866.52 MB 2025-02-14 15:26:27,260 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 15:26:27,261 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:27,262 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:26:27,262 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:27,262 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 15:26:27,267 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 15:26:27,268 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:27,268 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 15:26:27,268 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-14 15:26:27,269 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:27,269 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:26:27,269 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:27,269 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:26:27,275 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 15:26:27,276 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:27,276 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:26:27,276 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:27,276 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:26:27,276 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 15:26:27,277 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:27,277 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:26:27,277 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:27,277 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 15:26:27,277 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 15:26:27,278 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:27,278 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:26:27,283 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:27,283 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:26:27,284 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:27,284 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:26:27,285 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:27,285 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:26:27,287 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:27,287 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:26:36,247 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:36,248 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:26:36,254 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 15:26:36,257 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:36,257 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 153, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 15:26:36,258 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:36,258 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 153, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 15:26:38,757 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 15:26:38,757 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 15:26:38,757 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.49 seconds 2025-02-14 15:26:38,757 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:26:38,757 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17312.05 MB 2025-02-14 15:26:38,757 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 17853.50 MB 2025-02-14 15:26:38,757 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 541.46 MB 2025-02-14 15:26:38,757 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33220.98 MB 2025-02-14 15:26:38,757 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20080.23 MB 2025-02-14 15:26:38,757 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13140.75 MB 2025-02-14 15:26:38,757 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26784.22 MB 2025-02-14 15:26:38,769 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 15:26:38,769 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 15:26:38,769 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:26:38,769 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:26:38,769 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17853.50 MB 2025-02-14 15:26:38,769 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18073.70 MB 2025-02-14 15:26:38,769 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 220.20 MB 2025-02-14 15:26:38,769 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20080.23 MB 2025-02-14 15:26:38,769 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 21378.37 MB 2025-02-14 15:26:38,769 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1298.14 MB 2025-02-14 15:26:38,769 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19950.19 MB 2025-02-14 15:26:39,511 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 15:26:39,511 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 15:26:39,511 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.74 seconds 2025-02-14 15:26:39,511 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:26:39,511 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18073.70 MB 2025-02-14 15:26:39,511 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18268.79 MB 2025-02-14 15:26:39,511 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 195.08 MB 2025-02-14 15:26:39,511 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 21378.37 MB 2025-02-14 15:26:39,511 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 19943.92 MB 2025-02-14 15:26:39,511 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1434.45 MB 2025-02-14 15:26:39,511 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 22245.18 MB 2025-02-14 15:26:39,519 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 15:26:39,519 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 15:26:39,519 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 15:26:39,519 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:26:39,519 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18268.72 MB 2025-02-14 15:26:39,519 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 18962.95 MB 2025-02-14 15:26:39,519 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 694.24 MB 2025-02-14 15:26:39,519 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19943.92 MB 2025-02-14 15:26:39,519 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 20640.17 MB 2025-02-14 15:26:39,519 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 696.25 MB 2025-02-14 15:26:39,519 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 19483.87 MB 2025-02-14 15:26:39,610 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 15:26:39,610 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 15:26:39,610 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 15:26:39,610 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:26:39,610 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18962.95 MB 2025-02-14 15:26:39,610 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19786.88 MB 2025-02-14 15:26:39,610 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 823.92 MB 2025-02-14 15:26:39,610 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 20640.17 MB 2025-02-14 15:26:39,610 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22728.93 MB 2025-02-14 15:26:39,610 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2088.76 MB 2025-02-14 15:26:39,610 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21825.67 MB 2025-02-14 15:26:39,611 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 15:26:39,611 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 15:26:39,611 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 15:26:39,611 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:26:39,611 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 18268.72 MB 2025-02-14 15:26:39,611 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19786.88 MB 2025-02-14 15:26:39,611 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1518.16 MB 2025-02-14 15:26:39,611 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 19943.92 MB 2025-02-14 15:26:39,611 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22728.93 MB 2025-02-14 15:26:39,611 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2785.02 MB 2025-02-14 15:26:39,611 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21825.67 MB 2025-02-14 15:26:39,670 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 15:26:39,670 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 15:26:39,670 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 15:26:39,670 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:26:39,670 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20046.99 MB 2025-02-14 15:26:39,670 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20330.17 MB 2025-02-14 15:26:39,670 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 283.18 MB 2025-02-14 15:26:39,670 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22728.93 MB 2025-02-14 15:26:39,670 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22879.93 MB 2025-02-14 15:26:39,670 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 150.99 MB 2025-02-14 15:26:39,670 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20601.26 MB 2025-02-14 15:26:39,678 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 15:26:39,678 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 15:26:39,678 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:26:39,678 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:26:39,678 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20481.92 MB 2025-02-14 15:26:39,678 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20683.59 MB 2025-02-14 15:26:39,678 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 201.67 MB 2025-02-14 15:26:39,678 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22879.93 MB 2025-02-14 15:26:39,678 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22882.03 MB 2025-02-14 15:26:39,678 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-14 15:26:39,678 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20693.43 MB 2025-02-14 15:26:39,679 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 15:26:39,679 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 15:26:39,679 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.42 seconds 2025-02-14 15:26:39,679 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:26:39,679 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 16778.98 MB 2025-02-14 15:26:39,679 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20884.56 MB 2025-02-14 15:26:39,679 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4105.58 MB 2025-02-14 15:26:39,679 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33220.98 MB 2025-02-14 15:26:39,679 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22882.03 MB 2025-02-14 15:26:39,679 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10338.96 MB 2025-02-14 15:26:39,679 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20884.56 MB 2025-02-14 15:26:39,943 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 15:26:39,943 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 15:26:39,943 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 15:26:39,943 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:26:39,943 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20884.56 MB 2025-02-14 15:26:39,943 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20984.98 MB 2025-02-14 15:26:39,943 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.42 MB 2025-02-14 15:26:39,943 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22882.03 MB 2025-02-14 15:26:39,943 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 22882.03 MB 2025-02-14 15:26:39,943 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:26:39,943 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21587.49 MB 2025-02-14 15:26:39,961 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8158, cut from 8160 2025-02-14 15:26:39,961 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-14 15:26:39,967 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 15:26:39,967 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 15:26:39,967 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:26:39,967 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:26:39,967 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20984.98 MB 2025-02-14 15:26:39,967 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25177.42 MB 2025-02-14 15:26:39,967 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4192.43 MB 2025-02-14 15:26:39,967 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 22882.03 MB 2025-02-14 15:26:39,967 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33365.69 MB 2025-02-14 15:26:39,967 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10483.66 MB 2025-02-14 15:26:39,967 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29369.62 MB 2025-02-14 15:26:40,199 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7950] 2025-02-14 15:26:40,201 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:40,201 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:26:40,203 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:40,203 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 15:26:40,210 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 15:26:40,211 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:40,212 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 15:26:40,212 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-14 15:26:40,213 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:40,213 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:26:40,214 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:40,214 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:26:40,222 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 15:26:40,223 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:40,223 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:26:40,224 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:40,224 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:26:40,224 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 15:26:40,225 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:40,225 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:26:40,226 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:40,226 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 15:26:40,226 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 15:26:40,227 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:40,227 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:26:40,232 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:40,232 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:26:40,233 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:40,233 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:26:40,235 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:40,235 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:26:40,237 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:40,237 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:26:47,362 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:47,363 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:26:47,367 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 15:26:47,369 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:47,369 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 154, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 15:26:47,370 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:47,370 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 154, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 15:26:49,763 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 15:26:49,763 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 15:26:49,763 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.39 seconds 2025-02-14 15:26:49,763 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:26:49,763 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26494.12 MB 2025-02-14 15:26:49,763 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27039.12 MB 2025-02-14 15:26:49,763 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 545.00 MB 2025-02-14 15:26:49,763 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33365.69 MB 2025-02-14 15:26:49,763 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29838.28 MB 2025-02-14 15:26:49,763 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3527.41 MB 2025-02-14 15:26:49,763 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35966.30 MB 2025-02-14 15:26:49,775 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 15:26:49,775 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 15:26:49,775 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:26:49,775 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:26:49,775 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27039.12 MB 2025-02-14 15:26:49,775 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27268.05 MB 2025-02-14 15:26:49,775 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 228.93 MB 2025-02-14 15:26:49,775 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29838.28 MB 2025-02-14 15:26:49,775 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30366.76 MB 2025-02-14 15:26:49,775 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 528.48 MB 2025-02-14 15:26:49,775 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29167.43 MB 2025-02-14 15:26:50,501 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 15:26:50,501 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 15:26:50,501 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.72 seconds 2025-02-14 15:26:50,501 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:26:50,501 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27268.05 MB 2025-02-14 15:26:50,501 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27465.79 MB 2025-02-14 15:26:50,501 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 197.74 MB 2025-02-14 15:26:50,501 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30366.76 MB 2025-02-14 15:26:50,501 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29754.39 MB 2025-02-14 15:26:50,501 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -612.37 MB 2025-02-14 15:26:50,501 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31439.53 MB 2025-02-14 15:26:50,508 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 15:26:50,508 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 15:26:50,508 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 15:26:50,508 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:26:50,508 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27465.72 MB 2025-02-14 15:26:50,508 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28170.19 MB 2025-02-14 15:26:50,508 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 704.47 MB 2025-02-14 15:26:50,508 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29754.39 MB 2025-02-14 15:26:50,508 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30108.81 MB 2025-02-14 15:26:50,508 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 354.42 MB 2025-02-14 15:26:50,508 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28698.19 MB 2025-02-14 15:26:50,589 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 15:26:50,589 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 15:26:50,589 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 15:26:50,589 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:26:50,589 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28170.19 MB 2025-02-14 15:26:50,589 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19951.94 MB 2025-02-14 15:26:50,589 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8218.25 MB 2025-02-14 15:26:50,589 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30108.81 MB 2025-02-14 15:26:50,589 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30108.81 MB 2025-02-14 15:26:50,589 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:26:50,589 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28509.29 MB 2025-02-14 15:26:50,589 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 15:26:50,589 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 15:26:50,589 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 15:26:50,590 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:26:50,590 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27465.72 MB 2025-02-14 15:26:50,590 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 19951.94 MB 2025-02-14 15:26:50,590 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -7513.78 MB 2025-02-14 15:26:50,590 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29754.39 MB 2025-02-14 15:26:50,590 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30108.81 MB 2025-02-14 15:26:50,590 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 354.42 MB 2025-02-14 15:26:50,590 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28509.29 MB 2025-02-14 15:26:50,647 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 15:26:50,647 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 15:26:50,647 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-14 15:26:50,647 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:26:50,647 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20215.59 MB 2025-02-14 15:26:50,647 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20501.30 MB 2025-02-14 15:26:50,647 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 285.71 MB 2025-02-14 15:26:50,647 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30108.81 MB 2025-02-14 15:26:50,647 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30108.81 MB 2025-02-14 15:26:50,647 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:26:50,647 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20776.36 MB 2025-02-14 15:26:50,655 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 15:26:50,655 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 15:26:50,655 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:26:50,655 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:26:50,655 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 20655.11 MB 2025-02-14 15:26:50,655 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 20860.11 MB 2025-02-14 15:26:50,655 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.00 MB 2025-02-14 15:26:50,655 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30108.81 MB 2025-02-14 15:26:50,655 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30108.81 MB 2025-02-14 15:26:50,655 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:26:50,655 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 20872.05 MB 2025-02-14 15:26:50,656 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 15:26:50,656 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 15:26:50,656 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.28 seconds 2025-02-14 15:26:50,656 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:26:50,656 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25957.57 MB 2025-02-14 15:26:50,656 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21061.16 MB 2025-02-14 15:26:50,656 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4896.41 MB 2025-02-14 15:26:50,656 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33365.69 MB 2025-02-14 15:26:50,656 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30108.81 MB 2025-02-14 15:26:50,656 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3256.88 MB 2025-02-14 15:26:50,656 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21061.16 MB 2025-02-14 15:26:50,920 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 15:26:50,921 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 15:26:50,921 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 15:26:50,921 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:26:50,921 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 21061.16 MB 2025-02-14 15:26:50,921 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21161.61 MB 2025-02-14 15:26:50,921 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.45 MB 2025-02-14 15:26:50,921 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30108.81 MB 2025-02-14 15:26:50,921 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30108.81 MB 2025-02-14 15:26:50,921 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:26:50,921 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 21764.34 MB 2025-02-14 15:26:50,938 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8161, cut from 8163 2025-02-14 15:26:50,939 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2.'] 2025-02-14 15:26:50,944 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 15:26:50,944 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 15:26:50,944 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:26:50,945 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:26:50,945 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17500.74 MB 2025-02-14 15:26:50,945 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21695.04 MB 2025-02-14 15:26:50,945 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4194.31 MB 2025-02-14 15:26:50,945 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30108.81 MB 2025-02-14 15:26:50,945 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34303.12 MB 2025-02-14 15:26:50,945 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4194.30 MB 2025-02-14 15:26:50,945 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25889.35 MB 2025-02-14 15:26:51,110 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7953] 2025-02-14 15:26:51,112 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:51,112 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:26:51,113 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:51,113 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 15:26:51,117 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 15:26:51,118 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:51,118 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 15:26:51,118 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2.'] 2025-02-14 15:26:51,119 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:51,119 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:26:51,120 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:51,120 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:26:51,125 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 15:26:51,126 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:51,126 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:26:51,126 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:51,126 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:26:51,126 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 15:26:51,127 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:51,127 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:26:51,127 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:51,127 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 15:26:51,127 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 15:26:51,128 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:51,128 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:26:51,132 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:51,132 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:26:51,133 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:51,133 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:26:51,134 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:51,134 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:26:51,136 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:51,136 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:26:59,198 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:59,198 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:26:59,203 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 15:26:59,204 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:59,204 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 175, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 15:26:59,206 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:26:59,206 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 175, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 15:27:01,913 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 15:27:01,913 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 15:27:01,913 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.70 seconds 2025-02-14 15:27:01,913 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:27:01,913 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23158.08 MB 2025-02-14 15:27:01,913 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23777.39 MB 2025-02-14 15:27:01,913 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 619.32 MB 2025-02-14 15:27:01,913 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34303.12 MB 2025-02-14 15:27:01,913 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27883.73 MB 2025-02-14 15:27:01,913 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6419.38 MB 2025-02-14 15:27:01,913 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32630.25 MB 2025-02-14 15:27:01,928 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 15:27:01,928 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 15:27:01,928 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:27:01,928 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:27:01,928 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23777.39 MB 2025-02-14 15:27:01,928 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24077.38 MB 2025-02-14 15:27:01,928 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 299.99 MB 2025-02-14 15:27:01,928 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27883.73 MB 2025-02-14 15:27:01,928 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28504.49 MB 2025-02-14 15:27:01,928 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 620.76 MB 2025-02-14 15:27:01,928 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26288.53 MB 2025-02-14 15:27:02,771 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 15:27:02,772 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 15:27:02,772 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.84 seconds 2025-02-14 15:27:02,772 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:27:02,772 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24077.38 MB 2025-02-14 15:27:02,772 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24309.62 MB 2025-02-14 15:27:02,772 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 232.24 MB 2025-02-14 15:27:02,772 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28504.49 MB 2025-02-14 15:27:02,772 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28156.36 MB 2025-02-14 15:27:02,772 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -348.13 MB 2025-02-14 15:27:02,772 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28248.86 MB 2025-02-14 15:27:02,780 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 15:27:02,780 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 15:27:02,780 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:27:02,780 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:27:02,780 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24309.62 MB 2025-02-14 15:27:02,780 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25136.10 MB 2025-02-14 15:27:02,780 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 826.47 MB 2025-02-14 15:27:02,780 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28156.36 MB 2025-02-14 15:27:02,780 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28156.36 MB 2025-02-14 15:27:02,780 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:27:02,780 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25756.23 MB 2025-02-14 15:27:02,874 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 15:27:02,874 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 15:27:02,874 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 15:27:02,874 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:27:02,874 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25136.10 MB 2025-02-14 15:27:02,874 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26117.86 MB 2025-02-14 15:27:02,874 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 981.77 MB 2025-02-14 15:27:02,874 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28156.36 MB 2025-02-14 15:27:02,874 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30635.20 MB 2025-02-14 15:27:02,874 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2478.83 MB 2025-02-14 15:27:02,874 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28544.50 MB 2025-02-14 15:27:02,875 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 15:27:02,875 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 15:27:02,875 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 15:27:02,875 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:27:02,875 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24309.62 MB 2025-02-14 15:27:02,875 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26117.86 MB 2025-02-14 15:27:02,875 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1808.24 MB 2025-02-14 15:27:02,875 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28156.36 MB 2025-02-14 15:27:02,875 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30635.20 MB 2025-02-14 15:27:02,875 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2478.83 MB 2025-02-14 15:27:02,875 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28544.50 MB 2025-02-14 15:27:02,944 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 15:27:02,945 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 15:27:02,945 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 15:27:02,945 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:27:02,945 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26427.52 MB 2025-02-14 15:27:02,945 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26763.08 MB 2025-02-14 15:27:02,945 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 335.56 MB 2025-02-14 15:27:02,945 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30635.20 MB 2025-02-14 15:27:02,945 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30815.55 MB 2025-02-14 15:27:02,945 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 180.36 MB 2025-02-14 15:27:02,945 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27080.61 MB 2025-02-14 15:27:02,953 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 15:27:02,953 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 15:27:02,954 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:27:02,954 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:27:02,954 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26943.73 MB 2025-02-14 15:27:02,954 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27149.70 MB 2025-02-14 15:27:02,954 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.97 MB 2025-02-14 15:27:02,954 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30815.55 MB 2025-02-14 15:27:02,954 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30815.55 MB 2025-02-14 15:27:02,954 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:27:02,954 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27170.79 MB 2025-02-14 15:27:02,955 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 15:27:02,955 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 15:27:02,955 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.75 seconds 2025-02-14 15:27:02,955 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:27:02,955 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22548.36 MB 2025-02-14 15:27:02,955 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27350.43 MB 2025-02-14 15:27:02,955 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4802.06 MB 2025-02-14 15:27:02,955 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34303.12 MB 2025-02-14 15:27:02,955 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30815.55 MB 2025-02-14 15:27:02,955 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3487.56 MB 2025-02-14 15:27:02,955 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27350.43 MB 2025-02-14 15:27:03,219 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 15:27:03,219 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 15:27:03,219 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 15:27:03,219 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:27:03,219 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27350.43 MB 2025-02-14 15:27:03,219 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27450.72 MB 2025-02-14 15:27:03,219 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.29 MB 2025-02-14 15:27:03,219 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30815.55 MB 2025-02-14 15:27:03,219 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30815.55 MB 2025-02-14 15:27:03,219 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:27:03,219 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28052.49 MB 2025-02-14 15:27:03,237 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8148, cut from 8150 2025-02-14 15:27:03,237 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-14 15:27:03,243 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 15:27:03,243 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 15:27:03,243 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:27:03,243 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:27:03,244 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23213.61 MB 2025-02-14 15:27:03,244 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27401.63 MB 2025-02-14 15:27:03,244 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4188.01 MB 2025-02-14 15:27:03,244 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30815.55 MB 2025-02-14 15:27:03,244 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41286.63 MB 2025-02-14 15:27:03,244 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10471.08 MB 2025-02-14 15:27:03,244 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31588.41 MB 2025-02-14 15:27:03,396 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7940] 2025-02-14 15:27:03,397 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:27:03,397 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:27:03,398 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:27:03,398 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 15:27:03,402 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 15:27:03,403 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:27:03,403 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 15:27:03,403 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-14 15:27:03,404 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:27:03,404 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:27:03,405 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:27:03,405 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:27:03,410 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 15:27:03,411 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:27:03,411 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:27:03,411 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:27:03,411 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:27:03,411 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 15:27:03,412 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:27:03,412 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:27:03,412 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:27:03,412 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 15:27:03,412 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 15:27:03,413 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:27:03,413 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:27:03,416 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:27:03,416 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:27:03,417 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:27:03,417 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:27:03,418 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:27:03,418 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:27:03,419 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:27:03,419 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:27:13,735 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:27:13,735 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:27:13,740 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 15:27:13,741 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:27:13,741 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 166, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 15:27:13,742 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:27:13,742 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 166, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 15:27:16,317 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 15:27:16,317 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 15:27:16,318 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.57 seconds 2025-02-14 15:27:16,318 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:27:16,318 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23217.09 MB 2025-02-14 15:27:16,318 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23804.55 MB 2025-02-14 15:27:16,318 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 587.46 MB 2025-02-14 15:27:16,318 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41286.63 MB 2025-02-14 15:27:16,318 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27533.51 MB 2025-02-14 15:27:16,318 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13753.12 MB 2025-02-14 15:27:16,318 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32689.27 MB 2025-02-14 15:27:16,330 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 15:27:16,330 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 15:27:16,330 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:27:16,330 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:27:16,330 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23804.55 MB 2025-02-14 15:27:16,330 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23963.29 MB 2025-02-14 15:27:16,330 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 158.73 MB 2025-02-14 15:27:16,330 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27533.51 MB 2025-02-14 15:27:16,330 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28057.80 MB 2025-02-14 15:27:16,330 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 524.29 MB 2025-02-14 15:27:16,330 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25905.19 MB 2025-02-14 15:27:17,047 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 15:27:17,047 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 15:27:17,047 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.71 seconds 2025-02-14 15:27:17,048 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:27:17,048 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23963.29 MB 2025-02-14 15:27:17,048 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24159.70 MB 2025-02-14 15:27:17,048 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 196.41 MB 2025-02-14 15:27:17,048 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28057.80 MB 2025-02-14 15:27:17,048 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27709.67 MB 2025-02-14 15:27:17,048 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -348.13 MB 2025-02-14 15:27:17,048 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28134.76 MB 2025-02-14 15:27:17,054 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 15:27:17,054 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 15:27:17,054 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 15:27:17,055 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:27:17,055 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24159.63 MB 2025-02-14 15:27:17,055 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24858.59 MB 2025-02-14 15:27:17,055 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 698.96 MB 2025-02-14 15:27:17,055 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27709.67 MB 2025-02-14 15:27:17,055 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27709.67 MB 2025-02-14 15:27:17,055 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:27:17,055 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25383.05 MB 2025-02-14 15:27:17,133 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 15:27:17,134 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 15:27:17,134 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 15:27:17,134 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:27:17,134 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24858.59 MB 2025-02-14 15:27:17,134 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25688.12 MB 2025-02-14 15:27:17,134 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 829.53 MB 2025-02-14 15:27:17,134 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27709.67 MB 2025-02-14 15:27:17,134 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29460.79 MB 2025-02-14 15:27:17,134 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1751.12 MB 2025-02-14 15:27:17,134 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27743.66 MB 2025-02-14 15:27:17,134 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 15:27:17,134 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 15:27:17,134 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 15:27:17,134 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:27:17,134 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24159.63 MB 2025-02-14 15:27:17,134 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25688.12 MB 2025-02-14 15:27:17,134 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1528.49 MB 2025-02-14 15:27:17,134 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27709.67 MB 2025-02-14 15:27:17,134 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29460.79 MB 2025-02-14 15:27:17,134 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1751.12 MB 2025-02-14 15:27:17,134 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27743.66 MB 2025-02-14 15:27:17,193 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 15:27:17,193 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 15:27:17,193 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 15:27:17,193 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:27:17,193 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25950.00 MB 2025-02-14 15:27:17,193 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26233.79 MB 2025-02-14 15:27:17,193 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 283.79 MB 2025-02-14 15:27:17,193 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29460.79 MB 2025-02-14 15:27:17,193 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29611.79 MB 2025-02-14 15:27:17,193 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 150.99 MB 2025-02-14 15:27:17,193 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26506.56 MB 2025-02-14 15:27:17,201 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 15:27:17,201 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 15:27:17,201 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:27:17,201 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:27:17,201 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26386.57 MB 2025-02-14 15:27:17,201 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26589.84 MB 2025-02-14 15:27:17,201 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 203.27 MB 2025-02-14 15:27:17,201 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29611.79 MB 2025-02-14 15:27:17,201 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29611.79 MB 2025-02-14 15:27:17,201 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:27:17,201 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26602.48 MB 2025-02-14 15:27:17,202 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 15:27:17,202 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 15:27:17,202 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.46 seconds 2025-02-14 15:27:17,202 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:27:17,202 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22638.73 MB 2025-02-14 15:27:17,202 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26790.91 MB 2025-02-14 15:27:17,202 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4152.18 MB 2025-02-14 15:27:17,202 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41286.63 MB 2025-02-14 15:27:17,202 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29611.79 MB 2025-02-14 15:27:17,203 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11674.85 MB 2025-02-14 15:27:17,203 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26790.91 MB 2025-02-14 15:27:17,466 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 15:27:17,467 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 15:27:17,467 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 15:27:17,467 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:27:17,467 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26790.91 MB 2025-02-14 15:27:17,467 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26891.38 MB 2025-02-14 15:27:17,467 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.47 MB 2025-02-14 15:27:17,467 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29611.79 MB 2025-02-14 15:27:17,467 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29611.79 MB 2025-02-14 15:27:17,467 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:27:17,467 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27494.18 MB 2025-02-14 15:27:17,484 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 15:27:17,485 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2,'] 2025-02-14 15:27:17,490 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 15:27:17,490 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 15:27:17,490 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:27:17,490 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:27:17,490 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 17783.37 MB 2025-02-14 15:27:17,490 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 21977.85 MB 2025-02-14 15:27:17,490 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4194.49 MB 2025-02-14 15:27:17,490 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29611.79 MB 2025-02-14 15:27:17,490 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33808.19 MB 2025-02-14 15:27:17,490 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4196.40 MB 2025-02-14 15:27:17,490 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26172.16 MB 2025-02-14 15:27:17,642 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 15:27:17,643 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:27:17,643 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:27:17,644 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:27:17,644 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 15:27:17,648 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 15:27:17,649 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:27:17,649 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 15:27:17,650 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2,'] 2025-02-14 15:27:17,650 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:27:17,650 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:27:17,651 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:27:17,651 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:27:17,656 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 15:27:17,657 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:27:17,657 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:27:17,657 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:27:17,657 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:27:17,657 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 15:27:17,658 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:27:17,658 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:27:17,658 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:27:17,658 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 15:27:17,658 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 15:27:17,659 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:27:17,659 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:27:17,663 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:27:17,663 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:27:17,665 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:27:17,665 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:27:17,666 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:27:17,666 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:27:17,669 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:27:17,669 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:27:44,768 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:27:44,768 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:27:44,775 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 15:27:44,777 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:27:44,777 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 214, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 15:27:44,779 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:27:44,779 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 214, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 15:27:48,095 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 15:27:48,095 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 15:27:48,095 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.31 seconds 2025-02-14 15:27:48,095 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:27:48,095 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23712.65 MB 2025-02-14 15:27:48,095 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24469.98 MB 2025-02-14 15:27:48,095 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 757.33 MB 2025-02-14 15:27:48,096 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33808.19 MB 2025-02-14 15:27:48,096 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30863.79 MB 2025-02-14 15:27:48,096 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2944.40 MB 2025-02-14 15:27:48,096 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33410.51 MB 2025-02-14 15:27:48,110 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 15:27:48,110 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 15:27:48,110 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:27:48,110 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:27:48,110 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24469.98 MB 2025-02-14 15:27:48,111 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24801.73 MB 2025-02-14 15:27:48,111 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 331.75 MB 2025-02-14 15:27:48,111 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30863.79 MB 2025-02-14 15:27:48,111 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30863.79 MB 2025-02-14 15:27:48,111 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:27:48,111 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27426.84 MB 2025-02-14 15:27:49,114 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 15:27:49,115 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 15:27:49,115 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.00 seconds 2025-02-14 15:27:49,115 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:27:49,115 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24801.73 MB 2025-02-14 15:27:49,115 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25079.09 MB 2025-02-14 15:27:49,115 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 277.36 MB 2025-02-14 15:27:49,115 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30863.79 MB 2025-02-14 15:27:49,115 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30863.79 MB 2025-02-14 15:27:49,115 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:27:49,115 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29057.10 MB 2025-02-14 15:27:49,124 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 15:27:49,124 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 15:27:49,124 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:27:49,124 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:27:49,124 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25079.09 MB 2025-02-14 15:27:49,124 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26066.13 MB 2025-02-14 15:27:49,124 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 987.04 MB 2025-02-14 15:27:49,124 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30863.79 MB 2025-02-14 15:27:49,124 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30863.79 MB 2025-02-14 15:27:49,124 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:27:49,124 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26806.74 MB 2025-02-14 15:27:49,240 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 15:27:49,240 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 15:27:49,240 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 15:27:49,240 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:27:49,240 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26066.13 MB 2025-02-14 15:27:49,240 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27237.76 MB 2025-02-14 15:27:49,240 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1171.63 MB 2025-02-14 15:27:49,240 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30863.79 MB 2025-02-14 15:27:49,240 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32348.57 MB 2025-02-14 15:27:49,240 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1484.78 MB 2025-02-14 15:27:49,240 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30134.62 MB 2025-02-14 15:27:49,241 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 15:27:49,241 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 15:27:49,241 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 15:27:49,241 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:27:49,241 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25079.09 MB 2025-02-14 15:27:49,241 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27237.76 MB 2025-02-14 15:27:49,241 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2158.67 MB 2025-02-14 15:27:49,241 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30863.79 MB 2025-02-14 15:27:49,241 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32348.57 MB 2025-02-14 15:27:49,241 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1484.78 MB 2025-02-14 15:27:49,241 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30134.62 MB 2025-02-14 15:27:49,325 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 15:27:49,325 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 15:27:49,325 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 15:27:49,325 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:27:49,325 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27607.58 MB 2025-02-14 15:27:49,325 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28008.34 MB 2025-02-14 15:27:49,325 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 400.76 MB 2025-02-14 15:27:49,325 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32348.57 MB 2025-02-14 15:27:49,325 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32564.58 MB 2025-02-14 15:27:49,325 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 216.01 MB 2025-02-14 15:27:49,325 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28380.10 MB 2025-02-14 15:27:49,335 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 15:27:49,335 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 15:27:49,335 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:27:49,335 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:27:49,335 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28224.08 MB 2025-02-14 15:27:49,335 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28430.97 MB 2025-02-14 15:27:49,335 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.88 MB 2025-02-14 15:27:49,335 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32564.58 MB 2025-02-14 15:27:49,335 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32564.58 MB 2025-02-14 15:27:49,335 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:27:49,335 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28472.93 MB 2025-02-14 15:27:49,336 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 15:27:49,336 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 15:27:49,336 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.55 seconds 2025-02-14 15:27:49,337 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:27:49,337 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22967.05 MB 2025-02-14 15:27:49,337 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28632.04 MB 2025-02-14 15:27:49,337 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5664.99 MB 2025-02-14 15:27:49,337 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33808.19 MB 2025-02-14 15:27:49,337 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32564.58 MB 2025-02-14 15:27:49,337 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1243.61 MB 2025-02-14 15:27:49,337 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28632.04 MB 2025-02-14 15:27:49,599 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 15:27:49,600 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 15:27:49,600 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 15:27:49,600 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:27:49,600 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23622.44 MB 2025-02-14 15:27:49,600 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23723.07 MB 2025-02-14 15:27:49,600 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.62 MB 2025-02-14 15:27:49,600 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32564.58 MB 2025-02-14 15:27:49,600 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32564.58 MB 2025-02-14 15:27:49,600 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:27:49,600 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24325.87 MB 2025-02-14 15:27:49,618 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 15:27:49,618 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for the video is 2.'] 2025-02-14 15:27:49,624 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 15:27:49,624 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 15:27:49,624 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:27:49,624 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:27:49,624 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23723.07 MB 2025-02-14 15:27:49,624 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27917.55 MB 2025-02-14 15:27:49,624 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4194.49 MB 2025-02-14 15:27:49,624 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32564.58 MB 2025-02-14 15:27:49,624 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40955.28 MB 2025-02-14 15:27:49,624 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 15:27:49,624 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32111.86 MB 2025-02-14 15:27:49,774 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 15:27:49,775 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:27:49,775 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:27:49,776 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:27:49,776 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 15:27:49,781 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 15:27:49,782 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:27:49,782 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 15:27:49,782 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for the video is 2.'] 2025-02-14 15:27:49,782 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:27:49,782 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:27:49,783 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:27:49,783 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:27:49,788 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 15:27:49,789 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:27:49,789 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:27:49,790 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:27:49,790 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:27:49,790 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 15:27:49,790 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:27:49,790 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:27:49,791 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:27:49,791 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 15:27:49,791 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 15:27:49,791 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:27:49,791 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:27:49,794 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:27:49,794 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:27:49,795 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:27:49,795 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:27:49,796 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:27:49,796 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:27:49,798 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:27:49,798 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:27:58,729 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:27:58,729 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:27:58,734 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 15:27:58,735 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:27:58,735 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 680, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 15:27:58,736 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:27:58,736 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 680, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 15:28:09,276 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 15:28:09,276 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 15:28:09,277 - resource_logging.py:150 - __exit__ - DEBUG - Time: 10.53 seconds 2025-02-14 15:28:09,277 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:28:09,277 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27081.53 MB 2025-02-14 15:28:09,277 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29489.06 MB 2025-02-14 15:28:09,277 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2407.53 MB 2025-02-14 15:28:09,277 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40955.28 MB 2025-02-14 15:28:09,277 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36117.15 MB 2025-02-14 15:28:09,277 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4838.13 MB 2025-02-14 15:28:09,277 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38364.84 MB 2025-02-14 15:28:09,323 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 15:28:09,323 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 15:28:09,323 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-14 15:28:09,323 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:28:09,323 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29489.06 MB 2025-02-14 15:28:09,323 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28687.44 MB 2025-02-14 15:28:09,323 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -801.63 MB 2025-02-14 15:28:09,323 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36117.15 MB 2025-02-14 15:28:09,323 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42429.58 MB 2025-02-14 15:28:09,323 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6312.43 MB 2025-02-14 15:28:09,323 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38391.40 MB 2025-02-14 15:28:11,235 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 15:28:11,235 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 15:28:11,235 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 15:28:11,235 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:28:11,235 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28687.44 MB 2025-02-14 15:28:11,235 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29218.28 MB 2025-02-14 15:28:11,235 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 15:28:11,235 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42429.58 MB 2025-02-14 15:28:11,235 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35125.20 MB 2025-02-14 15:28:11,235 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7304.38 MB 2025-02-14 15:28:11,235 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33197.61 MB 2025-02-14 15:28:11,249 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 15:28:11,249 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 15:28:11,249 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:28:11,249 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:28:11,249 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29218.28 MB 2025-02-14 15:28:11,249 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31107.63 MB 2025-02-14 15:28:11,249 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.35 MB 2025-02-14 15:28:11,249 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35125.20 MB 2025-02-14 15:28:11,249 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37012.64 MB 2025-02-14 15:28:11,249 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 15:28:11,249 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32525.06 MB 2025-02-14 15:28:11,460 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 15:28:11,460 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 15:28:11,460 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 15:28:11,460 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:28:11,460 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31107.63 MB 2025-02-14 15:28:11,460 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33349.49 MB 2025-02-14 15:28:11,460 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 15:28:11,460 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37012.64 MB 2025-02-14 15:28:11,460 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42203.09 MB 2025-02-14 15:28:11,460 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 15:28:11,460 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38893.77 MB 2025-02-14 15:28:11,461 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 15:28:11,461 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 15:28:11,461 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 15:28:11,461 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:28:11,461 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29218.28 MB 2025-02-14 15:28:11,461 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33349.49 MB 2025-02-14 15:28:11,461 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.21 MB 2025-02-14 15:28:11,461 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35125.20 MB 2025-02-14 15:28:11,461 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42203.09 MB 2025-02-14 15:28:11,461 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7077.89 MB 2025-02-14 15:28:11,461 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38893.77 MB 2025-02-14 15:28:11,626 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 15:28:11,626 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 15:28:11,626 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 15:28:11,626 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:28:11,626 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34057.28 MB 2025-02-14 15:28:11,626 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34824.28 MB 2025-02-14 15:28:11,626 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 15:28:11,626 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42203.09 MB 2025-02-14 15:28:11,626 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42618.32 MB 2025-02-14 15:28:11,626 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 15:28:11,626 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35532.07 MB 2025-02-14 15:28:11,643 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 15:28:11,643 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 15:28:11,643 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:28:11,643 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:28:11,643 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35237.17 MB 2025-02-14 15:28:11,643 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35442.04 MB 2025-02-14 15:28:11,643 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 204.88 MB 2025-02-14 15:28:11,643 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42618.32 MB 2025-02-14 15:28:11,643 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42618.32 MB 2025-02-14 15:28:11,643 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:28:11,643 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35652.54 MB 2025-02-14 15:28:11,644 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 15:28:11,644 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 15:28:11,644 - resource_logging.py:150 - __exit__ - DEBUG - Time: 12.91 seconds 2025-02-14 15:28:11,644 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:28:11,644 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24712.36 MB 2025-02-14 15:28:11,644 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35641.99 MB 2025-02-14 15:28:11,644 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 10929.63 MB 2025-02-14 15:28:11,644 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40955.28 MB 2025-02-14 15:28:11,644 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42618.32 MB 2025-02-14 15:28:11,644 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1663.04 MB 2025-02-14 15:28:11,644 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35652.54 MB 2025-02-14 15:28:11,908 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 15:28:11,908 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 15:28:11,908 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 15:28:11,908 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:28:11,908 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35641.99 MB 2025-02-14 15:28:11,908 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35741.89 MB 2025-02-14 15:28:11,908 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 99.90 MB 2025-02-14 15:28:11,908 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42618.32 MB 2025-02-14 15:28:11,908 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42618.32 MB 2025-02-14 15:28:11,908 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:28:11,908 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36341.30 MB 2025-02-14 15:28:11,935 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8116, cut from 8118 2025-02-14 15:28:11,935 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-14 15:28:11,945 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 15:28:11,945 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 15:28:11,945 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-14 15:28:11,945 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:28:11,945 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25974.15 MB 2025-02-14 15:28:11,946 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30145.39 MB 2025-02-14 15:28:11,946 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4171.24 MB 2025-02-14 15:28:11,946 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42618.32 MB 2025-02-14 15:28:11,946 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50960.79 MB 2025-02-14 15:28:11,946 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8342.47 MB 2025-02-14 15:28:11,946 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34316.62 MB 2025-02-14 15:28:12,107 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7908] 2025-02-14 15:28:12,109 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:28:12,109 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:28:12,110 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:28:12,110 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 15:28:12,114 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 15:28:12,115 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:28:12,115 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 15:28:12,115 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-14 15:28:12,116 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:28:12,116 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:28:12,117 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:28:12,117 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:28:12,123 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 15:28:12,123 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:28:12,123 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:28:12,124 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:28:12,124 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:28:12,124 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 15:28:12,124 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:28:12,124 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:28:12,125 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:28:12,125 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 15:28:12,125 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 15:28:12,125 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:28:12,125 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:28:12,130 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:28:12,130 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:28:12,131 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:28:12,131 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:28:12,132 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:28:12,132 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:28:12,134 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:28:12,134 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:28:21,800 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:28:21,800 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:28:21,805 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 15:28:21,806 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:28:21,806 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 194, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 15:28:21,807 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:28:21,807 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 194, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 15:28:24,853 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 15:28:24,853 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 15:28:24,853 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.04 seconds 2025-02-14 15:28:24,853 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:28:24,854 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23816.73 MB 2025-02-14 15:28:24,854 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24503.29 MB 2025-02-14 15:28:24,854 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 686.56 MB 2025-02-14 15:28:24,854 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50960.79 MB 2025-02-14 15:28:24,854 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29987.18 MB 2025-02-14 15:28:24,854 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20973.62 MB 2025-02-14 15:28:24,854 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33515.40 MB 2025-02-14 15:28:24,867 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 15:28:24,867 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 15:28:24,867 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:28:24,867 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:28:24,867 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24503.29 MB 2025-02-14 15:28:24,868 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24828.83 MB 2025-02-14 15:28:24,868 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 325.55 MB 2025-02-14 15:28:24,868 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29987.18 MB 2025-02-14 15:28:24,868 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29987.18 MB 2025-02-14 15:28:24,868 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:28:24,868 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27221.25 MB 2025-02-14 15:28:25,802 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 15:28:25,802 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 15:28:25,802 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.93 seconds 2025-02-14 15:28:25,802 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:28:25,802 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24828.83 MB 2025-02-14 15:28:25,802 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25084.97 MB 2025-02-14 15:28:25,802 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 256.13 MB 2025-02-14 15:28:25,802 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29987.18 MB 2025-02-14 15:28:25,802 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29515.32 MB 2025-02-14 15:28:25,802 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -471.86 MB 2025-02-14 15:28:25,802 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29085.24 MB 2025-02-14 15:28:25,810 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 15:28:25,810 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 15:28:25,810 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:28:25,810 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:28:25,810 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25084.97 MB 2025-02-14 15:28:25,810 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25996.44 MB 2025-02-14 15:28:25,810 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 911.48 MB 2025-02-14 15:28:25,810 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29515.32 MB 2025-02-14 15:28:25,810 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29515.32 MB 2025-02-14 15:28:25,810 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:28:25,810 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26680.36 MB 2025-02-14 15:28:25,915 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 15:28:25,915 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 15:28:25,915 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 15:28:25,915 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:28:25,916 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25996.44 MB 2025-02-14 15:28:25,916 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27078.67 MB 2025-02-14 15:28:25,916 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1082.22 MB 2025-02-14 15:28:25,916 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29515.32 MB 2025-02-14 15:28:25,916 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31801.21 MB 2025-02-14 15:28:25,916 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2285.90 MB 2025-02-14 15:28:25,916 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29753.75 MB 2025-02-14 15:28:25,916 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 15:28:25,916 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 15:28:25,916 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 15:28:25,916 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:28:25,916 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25084.97 MB 2025-02-14 15:28:25,916 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27078.67 MB 2025-02-14 15:28:25,916 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1993.70 MB 2025-02-14 15:28:25,916 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29515.32 MB 2025-02-14 15:28:25,916 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31801.21 MB 2025-02-14 15:28:25,916 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2285.90 MB 2025-02-14 15:28:25,916 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29753.75 MB 2025-02-14 15:28:25,996 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 15:28:25,996 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 15:28:25,996 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 15:28:25,996 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:28:25,996 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27420.17 MB 2025-02-14 15:28:25,996 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27790.25 MB 2025-02-14 15:28:25,996 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 370.08 MB 2025-02-14 15:28:25,996 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31801.21 MB 2025-02-14 15:28:25,996 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32000.44 MB 2025-02-14 15:28:25,996 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 199.23 MB 2025-02-14 15:28:25,996 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28138.08 MB 2025-02-14 15:28:26,005 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 15:28:26,005 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 15:28:26,005 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:28:26,005 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:28:26,005 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27989.48 MB 2025-02-14 15:28:26,005 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28196.01 MB 2025-02-14 15:28:26,005 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.53 MB 2025-02-14 15:28:26,005 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32000.44 MB 2025-02-14 15:28:26,005 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32000.44 MB 2025-02-14 15:28:26,005 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:28:26,005 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28237.71 MB 2025-02-14 15:28:26,006 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 15:28:26,007 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 15:28:26,007 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.20 seconds 2025-02-14 15:28:26,007 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:28:26,007 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23140.82 MB 2025-02-14 15:28:26,007 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28397.08 MB 2025-02-14 15:28:26,007 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5256.26 MB 2025-02-14 15:28:26,007 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50960.79 MB 2025-02-14 15:28:26,007 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32000.44 MB 2025-02-14 15:28:26,007 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18960.35 MB 2025-02-14 15:28:26,007 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28397.08 MB 2025-02-14 15:28:26,269 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 15:28:26,269 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 15:28:26,269 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 15:28:26,269 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:28:26,269 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28397.08 MB 2025-02-14 15:28:26,269 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28497.54 MB 2025-02-14 15:28:26,269 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.47 MB 2025-02-14 15:28:26,269 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32000.44 MB 2025-02-14 15:28:26,269 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32000.44 MB 2025-02-14 15:28:26,269 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:28:26,269 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29100.34 MB 2025-02-14 15:28:26,287 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 15:28:26,288 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 15:28:26,294 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 15:28:26,294 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 15:28:26,294 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:28:26,294 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:28:26,294 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23854.20 MB 2025-02-14 15:28:26,294 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28048.69 MB 2025-02-14 15:28:26,294 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4194.49 MB 2025-02-14 15:28:26,294 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32000.44 MB 2025-02-14 15:28:26,294 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42490.40 MB 2025-02-14 15:28:26,294 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 15:28:26,294 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32242.99 MB 2025-02-14 15:28:26,459 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 15:28:26,461 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:28:26,461 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:28:26,462 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:28:26,462 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 15:28:26,468 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 15:28:26,469 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:28:26,469 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 15:28:26,469 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 15:28:26,470 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:28:26,470 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:28:26,471 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:28:26,471 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:28:26,477 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 15:28:26,477 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:28:26,477 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:28:26,478 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:28:26,478 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:28:26,478 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 15:28:26,478 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:28:26,478 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:28:26,479 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:28:26,479 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 15:28:26,479 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 15:28:26,479 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:28:26,479 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:28:26,484 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:28:26,484 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:28:26,486 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:28:26,486 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:28:26,487 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:28:26,487 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:28:26,490 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:28:26,490 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:29:08,178 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:29:08,178 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:29:08,183 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 15:29:08,184 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:29:08,185 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 195, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 15:29:08,185 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:29:08,185 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 195, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 15:29:11,202 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 15:29:11,202 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 15:29:11,202 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.01 seconds 2025-02-14 15:29:11,202 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:29:11,202 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23945.43 MB 2025-02-14 15:29:11,202 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24635.52 MB 2025-02-14 15:29:11,202 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 690.09 MB 2025-02-14 15:29:11,202 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42490.40 MB 2025-02-14 15:29:11,202 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28791.80 MB 2025-02-14 15:29:11,202 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13698.60 MB 2025-02-14 15:29:11,202 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33644.10 MB 2025-02-14 15:29:11,217 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 15:29:11,217 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 15:29:11,217 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:29:11,217 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:29:11,217 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24635.52 MB 2025-02-14 15:29:11,217 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24956.48 MB 2025-02-14 15:29:11,217 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 320.96 MB 2025-02-14 15:29:11,217 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28791.80 MB 2025-02-14 15:29:11,217 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29475.47 MB 2025-02-14 15:29:11,217 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 683.67 MB 2025-02-14 15:29:11,217 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27357.75 MB 2025-02-14 15:29:12,146 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 15:29:12,146 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 15:29:12,146 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.93 seconds 2025-02-14 15:29:12,146 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:29:12,146 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24956.48 MB 2025-02-14 15:29:12,146 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25212.61 MB 2025-02-14 15:29:12,146 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 256.13 MB 2025-02-14 15:29:12,146 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29475.47 MB 2025-02-14 15:29:12,146 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29475.47 MB 2025-02-14 15:29:12,146 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:29:12,146 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29212.89 MB 2025-02-14 15:29:12,154 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 15:29:12,154 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 15:29:12,154 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:29:12,154 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:29:12,154 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25212.55 MB 2025-02-14 15:29:12,154 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26124.02 MB 2025-02-14 15:29:12,154 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 911.48 MB 2025-02-14 15:29:12,154 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29475.47 MB 2025-02-14 15:29:12,154 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29475.47 MB 2025-02-14 15:29:12,154 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:29:12,154 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26807.94 MB 2025-02-14 15:29:12,259 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 15:29:12,259 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 15:29:12,259 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 15:29:12,259 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:29:12,259 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26124.02 MB 2025-02-14 15:29:12,259 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27206.25 MB 2025-02-14 15:29:12,259 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1082.22 MB 2025-02-14 15:29:12,259 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29475.47 MB 2025-02-14 15:29:12,259 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32218.55 MB 2025-02-14 15:29:12,259 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2743.07 MB 2025-02-14 15:29:12,259 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29883.16 MB 2025-02-14 15:29:12,260 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 15:29:12,260 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 15:29:12,260 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 15:29:12,260 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:29:12,260 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25212.55 MB 2025-02-14 15:29:12,260 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27206.25 MB 2025-02-14 15:29:12,260 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1993.70 MB 2025-02-14 15:29:12,260 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29475.47 MB 2025-02-14 15:29:12,260 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32218.55 MB 2025-02-14 15:29:12,260 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2743.07 MB 2025-02-14 15:29:12,260 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29883.16 MB 2025-02-14 15:29:12,340 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 15:29:12,340 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 15:29:12,340 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 15:29:12,340 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:29:12,340 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27547.75 MB 2025-02-14 15:29:12,340 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27919.67 MB 2025-02-14 15:29:12,340 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 371.91 MB 2025-02-14 15:29:12,340 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32218.55 MB 2025-02-14 15:29:12,340 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32417.78 MB 2025-02-14 15:29:12,340 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 199.23 MB 2025-02-14 15:29:12,340 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28267.14 MB 2025-02-14 15:29:12,349 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 15:29:12,349 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 15:29:12,349 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:29:12,349 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:29:12,349 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28118.89 MB 2025-02-14 15:29:12,350 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28325.90 MB 2025-02-14 15:29:12,350 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 207.01 MB 2025-02-14 15:29:12,350 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32417.78 MB 2025-02-14 15:29:12,350 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32417.78 MB 2025-02-14 15:29:12,350 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:29:12,350 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28361.50 MB 2025-02-14 15:29:12,351 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 15:29:12,351 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 15:29:12,351 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.16 seconds 2025-02-14 15:29:12,351 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:29:12,351 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23266.03 MB 2025-02-14 15:29:12,351 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28526.97 MB 2025-02-14 15:29:12,351 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5260.94 MB 2025-02-14 15:29:12,351 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42490.40 MB 2025-02-14 15:29:12,351 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32417.78 MB 2025-02-14 15:29:12,351 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10072.62 MB 2025-02-14 15:29:12,351 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28526.97 MB 2025-02-14 15:29:12,615 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 15:29:12,615 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 15:29:12,615 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 15:29:12,615 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:29:12,615 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28526.97 MB 2025-02-14 15:29:12,615 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28627.44 MB 2025-02-14 15:29:12,615 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.47 MB 2025-02-14 15:29:12,615 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32417.78 MB 2025-02-14 15:29:12,615 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32417.78 MB 2025-02-14 15:29:12,615 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:29:12,615 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29230.24 MB 2025-02-14 15:29:12,633 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 15:29:12,633 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2,'] 2025-02-14 15:29:12,639 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 15:29:12,639 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 15:29:12,639 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:29:12,639 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:29:12,639 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23981.25 MB 2025-02-14 15:29:12,640 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28175.73 MB 2025-02-14 15:29:12,640 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4194.49 MB 2025-02-14 15:29:12,640 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32417.78 MB 2025-02-14 15:29:12,640 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42907.73 MB 2025-02-14 15:29:12,640 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 15:29:12,640 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32370.04 MB 2025-02-14 15:29:12,802 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 15:29:12,803 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:29:12,803 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:29:12,804 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:29:12,804 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 15:29:12,809 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 15:29:12,810 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:29:12,810 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 15:29:12,810 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2,'] 2025-02-14 15:29:12,810 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:29:12,810 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:29:12,811 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:29:12,811 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:29:12,817 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 15:29:12,817 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:29:12,817 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:29:12,818 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:29:12,818 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:29:12,818 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 15:29:12,818 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:29:12,818 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:29:12,819 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:29:12,819 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 15:29:12,819 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 15:29:12,819 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:29:12,819 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:29:12,824 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:29:12,824 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:29:12,826 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:29:12,826 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:29:12,827 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:29:12,827 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:29:12,830 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:29:12,830 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:30:03,934 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:30:03,934 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:30:03,939 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 15:30:03,940 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:30:03,940 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 953, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 15:30:03,941 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:30:03,941 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 953, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 15:30:18,522 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 15:30:18,522 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 15:30:18,522 - resource_logging.py:150 - __exit__ - DEBUG - Time: 14.58 seconds 2025-02-14 15:30:18,522 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:30:18,522 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29349.02 MB 2025-02-14 15:30:18,522 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32721.63 MB 2025-02-14 15:30:18,522 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3372.61 MB 2025-02-14 15:30:18,522 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42907.73 MB 2025-02-14 15:30:18,522 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41284.53 MB 2025-02-14 15:30:18,522 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1623.20 MB 2025-02-14 15:30:18,522 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41538.30 MB 2025-02-14 15:30:18,583 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 15:30:18,583 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 15:30:18,583 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 15:30:18,583 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:30:18,584 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32721.63 MB 2025-02-14 15:30:18,584 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30471.86 MB 2025-02-14 15:30:18,584 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2249.78 MB 2025-02-14 15:30:18,584 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41284.53 MB 2025-02-14 15:30:18,584 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47909.44 MB 2025-02-14 15:30:18,584 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6624.90 MB 2025-02-14 15:30:18,584 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43601.73 MB 2025-02-14 15:30:20,490 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 15:30:20,490 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 15:30:20,490 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 15:30:20,490 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:30:20,491 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30471.86 MB 2025-02-14 15:30:20,491 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31002.70 MB 2025-02-14 15:30:20,491 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 15:30:20,491 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47909.44 MB 2025-02-14 15:30:20,491 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37910.22 MB 2025-02-14 15:30:20,491 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9999.22 MB 2025-02-14 15:30:20,491 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34982.03 MB 2025-02-14 15:30:20,505 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 15:30:20,505 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 15:30:20,505 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:30:20,505 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:30:20,505 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31002.70 MB 2025-02-14 15:30:20,505 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32892.05 MB 2025-02-14 15:30:20,505 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.35 MB 2025-02-14 15:30:20,505 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37910.22 MB 2025-02-14 15:30:20,505 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38853.94 MB 2025-02-14 15:30:20,505 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 15:30:20,505 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34309.48 MB 2025-02-14 15:30:20,714 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 15:30:20,714 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 15:30:20,714 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 15:30:20,714 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:30:20,714 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32892.05 MB 2025-02-14 15:30:20,714 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35133.91 MB 2025-02-14 15:30:20,714 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 15:30:20,714 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38853.94 MB 2025-02-14 15:30:20,714 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44044.39 MB 2025-02-14 15:30:20,714 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 15:30:20,714 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40678.19 MB 2025-02-14 15:30:20,714 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 15:30:20,714 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 15:30:20,714 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 15:30:20,714 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:30:20,714 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31002.70 MB 2025-02-14 15:30:20,714 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35133.91 MB 2025-02-14 15:30:20,714 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.21 MB 2025-02-14 15:30:20,714 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37910.22 MB 2025-02-14 15:30:20,714 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44044.39 MB 2025-02-14 15:30:20,714 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 15:30:20,715 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40678.19 MB 2025-02-14 15:30:20,880 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 15:30:20,880 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 15:30:20,880 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 15:30:20,880 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:30:20,880 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35841.70 MB 2025-02-14 15:30:20,880 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36608.70 MB 2025-02-14 15:30:20,880 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 15:30:20,880 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44044.39 MB 2025-02-14 15:30:20,880 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44459.62 MB 2025-02-14 15:30:20,880 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 15:30:20,880 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37316.49 MB 2025-02-14 15:30:20,898 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 15:30:20,898 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 15:30:20,898 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:30:20,898 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:30:20,898 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37021.59 MB 2025-02-14 15:30:20,898 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37227.47 MB 2025-02-14 15:30:20,898 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.88 MB 2025-02-14 15:30:20,898 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44459.62 MB 2025-02-14 15:30:20,898 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44459.62 MB 2025-02-14 15:30:20,898 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:30:20,898 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37445.78 MB 2025-02-14 15:30:20,899 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 15:30:20,899 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 15:30:20,899 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.96 seconds 2025-02-14 15:30:20,899 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:30:20,899 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26028.69 MB 2025-02-14 15:30:20,899 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37427.83 MB 2025-02-14 15:30:20,899 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11399.14 MB 2025-02-14 15:30:20,899 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42907.73 MB 2025-02-14 15:30:20,900 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44459.62 MB 2025-02-14 15:30:20,900 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1551.89 MB 2025-02-14 15:30:20,900 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37445.78 MB 2025-02-14 15:30:21,162 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 15:30:21,162 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 15:30:21,162 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 15:30:21,162 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:30:21,162 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37427.83 MB 2025-02-14 15:30:21,162 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37527.94 MB 2025-02-14 15:30:21,162 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.11 MB 2025-02-14 15:30:21,162 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44459.62 MB 2025-02-14 15:30:21,162 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44459.62 MB 2025-02-14 15:30:21,162 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:30:21,162 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38128.60 MB 2025-02-14 15:30:21,180 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8133, cut from 8135 2025-02-14 15:30:21,180 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 15:30:21,186 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 15:30:21,186 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 15:30:21,186 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:30:21,186 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:30:21,186 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27290.90 MB 2025-02-14 15:30:21,186 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31470.51 MB 2025-02-14 15:30:21,186 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4179.61 MB 2025-02-14 15:30:21,186 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44459.62 MB 2025-02-14 15:30:21,186 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48639.25 MB 2025-02-14 15:30:21,186 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4179.62 MB 2025-02-14 15:30:21,186 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35650.13 MB 2025-02-14 15:30:21,348 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7925] 2025-02-14 15:30:21,350 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:30:21,350 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:30:21,351 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:30:21,351 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 15:30:21,355 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 15:30:21,356 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:30:21,356 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 15:30:21,356 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 15:30:21,357 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:30:21,357 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:30:21,358 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:30:21,358 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:30:21,364 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 15:30:21,364 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:30:21,364 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:30:21,365 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:30:21,365 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:30:21,365 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 15:30:21,365 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:30:21,365 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:30:21,366 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:30:21,366 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 15:30:21,366 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 15:30:21,366 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:30:21,366 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:30:21,371 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:30:21,372 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:30:21,373 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:30:21,373 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:30:21,374 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:30:21,374 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:30:21,377 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:30:21,377 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:30:31,454 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:30:31,454 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:30:31,459 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 15:30:31,460 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:30:31,460 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1140, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 15:30:31,461 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:30:31,461 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1140, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 15:30:49,177 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 15:30:49,177 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 15:30:49,177 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.71 seconds 2025-02-14 15:30:49,177 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:30:49,177 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30773.79 MB 2025-02-14 15:30:49,177 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34808.71 MB 2025-02-14 15:30:49,177 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4034.92 MB 2025-02-14 15:30:49,177 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48639.25 MB 2025-02-14 15:30:49,177 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41928.36 MB 2025-02-14 15:30:49,177 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6710.89 MB 2025-02-14 15:30:49,177 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43643.36 MB 2025-02-14 15:30:49,250 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 15:30:49,250 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 15:30:49,250 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 15:30:49,250 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:30:49,250 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34808.71 MB 2025-02-14 15:30:49,250 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31565.74 MB 2025-02-14 15:30:49,250 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3242.97 MB 2025-02-14 15:30:49,250 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41928.36 MB 2025-02-14 15:30:49,250 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52686.75 MB 2025-02-14 15:30:49,250 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10758.39 MB 2025-02-14 15:30:49,250 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47034.98 MB 2025-02-14 15:30:51,173 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 15:30:51,173 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 15:30:51,173 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 15:30:51,173 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:30:51,173 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31565.74 MB 2025-02-14 15:30:51,173 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32096.58 MB 2025-02-14 15:30:51,173 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 15:30:51,173 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52686.75 MB 2025-02-14 15:30:51,173 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37209.77 MB 2025-02-14 15:30:51,173 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15476.98 MB 2025-02-14 15:30:51,173 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36075.91 MB 2025-02-14 15:30:51,188 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 15:30:51,188 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 15:30:51,188 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:30:51,188 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:30:51,188 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32096.58 MB 2025-02-14 15:30:51,188 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33985.93 MB 2025-02-14 15:30:51,188 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.35 MB 2025-02-14 15:30:51,188 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37209.77 MB 2025-02-14 15:30:51,188 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39097.20 MB 2025-02-14 15:30:51,188 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 15:30:51,188 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35403.36 MB 2025-02-14 15:30:51,401 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 15:30:51,401 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 15:30:51,401 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 15:30:51,401 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:30:51,401 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33985.93 MB 2025-02-14 15:30:51,401 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36227.79 MB 2025-02-14 15:30:51,401 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 15:30:51,401 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39097.20 MB 2025-02-14 15:30:51,401 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44759.52 MB 2025-02-14 15:30:51,401 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 15:30:51,401 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41772.07 MB 2025-02-14 15:30:51,402 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 15:30:51,402 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 15:30:51,402 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 15:30:51,402 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:30:51,402 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32096.58 MB 2025-02-14 15:30:51,402 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36227.79 MB 2025-02-14 15:30:51,402 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.21 MB 2025-02-14 15:30:51,402 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37209.77 MB 2025-02-14 15:30:51,402 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44759.52 MB 2025-02-14 15:30:51,402 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-14 15:30:51,402 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41772.07 MB 2025-02-14 15:30:51,566 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 15:30:51,566 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 15:30:51,566 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 15:30:51,566 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:30:51,566 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36935.58 MB 2025-02-14 15:30:51,566 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37702.58 MB 2025-02-14 15:30:51,566 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 15:30:51,566 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44759.52 MB 2025-02-14 15:30:51,566 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45176.85 MB 2025-02-14 15:30:51,566 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 15:30:51,566 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38410.37 MB 2025-02-14 15:30:51,583 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 15:30:51,583 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 15:30:51,583 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:30:51,583 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:30:51,583 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38115.47 MB 2025-02-14 15:30:51,583 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38321.43 MB 2025-02-14 15:30:51,583 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.96 MB 2025-02-14 15:30:51,584 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45176.85 MB 2025-02-14 15:30:51,584 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45176.85 MB 2025-02-14 15:30:51,584 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:30:51,584 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38537.41 MB 2025-02-14 15:30:51,585 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 15:30:51,585 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 15:30:51,585 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.12 seconds 2025-02-14 15:30:51,585 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:30:51,585 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26801.94 MB 2025-02-14 15:30:51,585 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38521.86 MB 2025-02-14 15:30:51,585 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11719.92 MB 2025-02-14 15:30:51,585 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48639.25 MB 2025-02-14 15:30:51,585 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45176.85 MB 2025-02-14 15:30:51,585 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3462.40 MB 2025-02-14 15:30:51,585 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38537.41 MB 2025-02-14 15:30:51,850 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 15:30:51,850 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 15:30:51,850 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 15:30:51,850 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:30:51,850 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38521.86 MB 2025-02-14 15:30:51,850 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38622.01 MB 2025-02-14 15:30:51,850 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.15 MB 2025-02-14 15:30:51,850 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45176.85 MB 2025-02-14 15:30:51,850 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45176.85 MB 2025-02-14 15:30:51,850 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:30:51,850 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39222.89 MB 2025-02-14 15:30:51,868 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8136, cut from 8138 2025-02-14 15:30:51,868 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 15:30:51,874 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 15:30:51,874 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 15:30:51,874 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:30:51,874 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:30:51,874 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28064.22 MB 2025-02-14 15:30:51,874 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32245.94 MB 2025-02-14 15:30:51,874 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4181.72 MB 2025-02-14 15:30:51,874 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45176.85 MB 2025-02-14 15:30:51,874 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55631.15 MB 2025-02-14 15:30:51,874 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10454.30 MB 2025-02-14 15:30:51,874 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36426.58 MB 2025-02-14 15:30:52,037 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7928] 2025-02-14 15:30:52,039 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:30:52,039 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:30:52,040 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:30:52,040 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 15:30:52,044 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 15:30:52,045 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:30:52,045 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 15:30:52,045 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 15:30:52,046 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:30:52,046 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:30:52,047 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:30:52,047 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:30:52,053 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 15:30:52,053 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:30:52,053 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:30:52,054 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:30:52,054 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:30:52,054 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 15:30:52,054 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:30:52,054 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:30:52,055 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:30:52,055 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 15:30:52,055 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 15:30:52,055 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:30:52,055 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:30:52,060 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:30:52,060 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:30:52,061 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:30:52,062 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:30:52,063 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:30:52,063 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:30:52,066 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:30:52,066 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:31:37,368 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:31:37,368 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:31:37,373 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 15:31:37,375 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:31:37,375 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 206, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 15:31:37,375 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:31:37,376 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 206, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 15:31:40,591 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 15:31:40,591 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 15:31:40,591 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.21 seconds 2025-02-14 15:31:40,591 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:31:40,591 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24387.26 MB 2025-02-14 15:31:40,591 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25116.28 MB 2025-02-14 15:31:40,591 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 729.02 MB 2025-02-14 15:31:40,591 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55631.15 MB 2025-02-14 15:31:40,591 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28363.98 MB 2025-02-14 15:31:40,591 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27267.17 MB 2025-02-14 15:31:40,591 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34085.93 MB 2025-02-14 15:31:40,606 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 15:31:40,606 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 15:31:40,606 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:31:40,606 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:31:40,607 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25116.28 MB 2025-02-14 15:31:40,607 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25379.60 MB 2025-02-14 15:31:40,607 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 263.32 MB 2025-02-14 15:31:40,607 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28363.98 MB 2025-02-14 15:31:40,607 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29720.84 MB 2025-02-14 15:31:40,607 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1356.86 MB 2025-02-14 15:31:40,607 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27878.19 MB 2025-02-14 15:31:41,535 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 15:31:41,535 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 15:31:41,535 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.93 seconds 2025-02-14 15:31:41,535 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:31:41,535 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25379.60 MB 2025-02-14 15:31:41,535 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25635.73 MB 2025-02-14 15:31:41,535 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 256.13 MB 2025-02-14 15:31:41,535 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29720.84 MB 2025-02-14 15:31:41,535 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29720.84 MB 2025-02-14 15:31:41,535 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:31:41,535 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29636.01 MB 2025-02-14 15:31:41,543 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 15:31:41,543 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 15:31:41,543 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:31:41,543 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:31:41,543 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25635.67 MB 2025-02-14 15:31:41,543 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26547.15 MB 2025-02-14 15:31:41,543 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 911.48 MB 2025-02-14 15:31:41,543 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29720.84 MB 2025-02-14 15:31:41,543 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29720.84 MB 2025-02-14 15:31:41,543 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:31:41,543 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27231.06 MB 2025-02-14 15:31:41,649 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 15:31:41,649 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 15:31:41,649 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 15:31:41,649 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:31:41,649 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26547.15 MB 2025-02-14 15:31:41,649 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27629.37 MB 2025-02-14 15:31:41,649 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1082.22 MB 2025-02-14 15:31:41,649 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29720.84 MB 2025-02-14 15:31:41,649 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32006.73 MB 2025-02-14 15:31:41,649 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2285.90 MB 2025-02-14 15:31:41,649 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30304.45 MB 2025-02-14 15:31:41,649 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 15:31:41,649 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 15:31:41,649 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 15:31:41,650 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:31:41,650 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25635.67 MB 2025-02-14 15:31:41,650 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27629.37 MB 2025-02-14 15:31:41,650 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1993.70 MB 2025-02-14 15:31:41,650 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29720.84 MB 2025-02-14 15:31:41,650 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32006.73 MB 2025-02-14 15:31:41,650 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2285.90 MB 2025-02-14 15:31:41,650 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30304.45 MB 2025-02-14 15:31:41,735 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 15:31:41,735 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 15:31:41,735 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 15:31:41,735 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:31:41,735 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27970.88 MB 2025-02-14 15:31:41,735 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28340.95 MB 2025-02-14 15:31:41,735 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 370.08 MB 2025-02-14 15:31:41,735 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32006.73 MB 2025-02-14 15:31:41,735 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32205.96 MB 2025-02-14 15:31:41,735 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 199.23 MB 2025-02-14 15:31:41,735 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28689.29 MB 2025-02-14 15:31:41,745 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 15:31:41,745 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 15:31:41,745 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:31:41,745 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:31:41,745 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28540.18 MB 2025-02-14 15:31:41,745 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28747.26 MB 2025-02-14 15:31:41,745 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 207.08 MB 2025-02-14 15:31:41,745 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32205.96 MB 2025-02-14 15:31:41,745 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32205.96 MB 2025-02-14 15:31:41,745 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:31:41,745 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28784.28 MB 2025-02-14 15:31:41,746 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 15:31:41,746 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 15:31:41,746 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.37 seconds 2025-02-14 15:31:41,746 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:31:41,746 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23669.54 MB 2025-02-14 15:31:41,746 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28948.33 MB 2025-02-14 15:31:41,746 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5278.80 MB 2025-02-14 15:31:41,746 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55631.15 MB 2025-02-14 15:31:41,746 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32205.96 MB 2025-02-14 15:31:41,746 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23425.19 MB 2025-02-14 15:31:41,746 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28948.33 MB 2025-02-14 15:31:42,009 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 15:31:42,009 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 15:31:42,009 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 15:31:42,009 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:31:42,009 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28948.33 MB 2025-02-14 15:31:42,009 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29048.80 MB 2025-02-14 15:31:42,009 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.47 MB 2025-02-14 15:31:42,009 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32205.96 MB 2025-02-14 15:31:42,009 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32205.96 MB 2025-02-14 15:31:42,009 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:31:42,009 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29651.60 MB 2025-02-14 15:31:42,027 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 15:31:42,027 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-14 15:31:42,033 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 15:31:42,033 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 15:31:42,033 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:31:42,033 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:31:42,033 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24382.92 MB 2025-02-14 15:31:42,033 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28577.40 MB 2025-02-14 15:31:42,033 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4194.49 MB 2025-02-14 15:31:42,033 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32205.96 MB 2025-02-14 15:31:42,033 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42695.92 MB 2025-02-14 15:31:42,033 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 15:31:42,033 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32771.71 MB 2025-02-14 15:31:42,197 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 15:31:42,198 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:31:42,198 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:31:42,199 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:31:42,199 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 15:31:42,204 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 15:31:42,205 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:31:42,205 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 15:31:42,205 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-14 15:31:42,206 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:31:42,206 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:31:42,206 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:31:42,206 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:31:42,212 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 15:31:42,213 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:31:42,213 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:31:42,213 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:31:42,213 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:31:42,213 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 15:31:42,214 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:31:42,214 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:31:42,214 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:31:42,214 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 15:31:42,214 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 15:31:42,215 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:31:42,215 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:31:42,220 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:31:42,220 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:31:42,221 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:31:42,221 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:31:42,222 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:31:42,222 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:31:42,225 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:31:42,225 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:32:31,428 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:32:31,429 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:32:31,433 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 15:32:31,435 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:32:31,435 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1065, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 15:32:31,436 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:32:31,436 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1065, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 15:32:47,793 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 15:32:47,793 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 15:32:47,793 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.35 seconds 2025-02-14 15:32:47,793 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:32:47,793 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30494.68 MB 2025-02-14 15:32:47,793 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34263.66 MB 2025-02-14 15:32:47,793 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3768.98 MB 2025-02-14 15:32:47,793 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42695.92 MB 2025-02-14 15:32:47,793 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41680.90 MB 2025-02-14 15:32:47,793 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1015.02 MB 2025-02-14 15:32:47,793 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43136.95 MB 2025-02-14 15:32:47,858 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 15:32:47,858 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 15:32:47,858 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 15:32:47,859 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:32:47,859 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34263.66 MB 2025-02-14 15:32:47,859 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31419.34 MB 2025-02-14 15:32:47,859 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2844.32 MB 2025-02-14 15:32:47,859 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41680.90 MB 2025-02-14 15:32:47,859 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50834.96 MB 2025-02-14 15:32:47,859 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9154.07 MB 2025-02-14 15:32:47,859 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45936.61 MB 2025-02-14 15:32:49,778 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 15:32:49,779 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 15:32:49,779 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 15:32:49,779 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:32:49,779 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31419.34 MB 2025-02-14 15:32:49,779 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31950.18 MB 2025-02-14 15:32:49,779 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 15:32:49,779 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50834.96 MB 2025-02-14 15:32:49,779 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39325.79 MB 2025-02-14 15:32:49,779 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -11509.17 MB 2025-02-14 15:32:49,779 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35929.51 MB 2025-02-14 15:32:49,793 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 15:32:49,793 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 15:32:49,793 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:32:49,793 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:32:49,793 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31950.18 MB 2025-02-14 15:32:49,793 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33839.53 MB 2025-02-14 15:32:49,793 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.35 MB 2025-02-14 15:32:49,793 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39325.79 MB 2025-02-14 15:32:49,793 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39325.79 MB 2025-02-14 15:32:49,793 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:32:49,793 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35256.96 MB 2025-02-14 15:32:50,000 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 15:32:50,000 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 15:32:50,000 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 15:32:50,000 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:32:50,000 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33839.53 MB 2025-02-14 15:32:50,000 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36081.39 MB 2025-02-14 15:32:50,000 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 15:32:50,000 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39325.79 MB 2025-02-14 15:32:50,000 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44516.25 MB 2025-02-14 15:32:50,000 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 15:32:50,000 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41625.67 MB 2025-02-14 15:32:50,001 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 15:32:50,001 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 15:32:50,001 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 15:32:50,001 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:32:50,001 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31950.18 MB 2025-02-14 15:32:50,001 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36081.39 MB 2025-02-14 15:32:50,001 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.21 MB 2025-02-14 15:32:50,001 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39325.79 MB 2025-02-14 15:32:50,001 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44516.25 MB 2025-02-14 15:32:50,001 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 15:32:50,001 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41625.67 MB 2025-02-14 15:32:50,159 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 15:32:50,159 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 15:32:50,159 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 15:32:50,159 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:32:50,159 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36789.18 MB 2025-02-14 15:32:50,159 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37556.18 MB 2025-02-14 15:32:50,159 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 15:32:50,159 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44516.25 MB 2025-02-14 15:32:50,159 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44931.48 MB 2025-02-14 15:32:50,159 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 15:32:50,159 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38263.97 MB 2025-02-14 15:32:50,176 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 15:32:50,176 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 15:32:50,176 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:32:50,176 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:32:50,176 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37969.07 MB 2025-02-14 15:32:50,176 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38174.80 MB 2025-02-14 15:32:50,176 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.74 MB 2025-02-14 15:32:50,176 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44931.48 MB 2025-02-14 15:32:50,176 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44931.48 MB 2025-02-14 15:32:50,176 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:32:50,176 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38395.48 MB 2025-02-14 15:32:50,177 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 15:32:50,177 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 15:32:50,177 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.74 seconds 2025-02-14 15:32:50,177 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:32:50,177 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26784.14 MB 2025-02-14 15:32:50,177 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38374.77 MB 2025-02-14 15:32:50,177 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11590.64 MB 2025-02-14 15:32:50,177 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42695.92 MB 2025-02-14 15:32:50,177 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44931.48 MB 2025-02-14 15:32:50,177 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2235.56 MB 2025-02-14 15:32:50,177 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38395.48 MB 2025-02-14 15:32:50,441 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 15:32:50,441 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 15:32:50,441 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 15:32:50,441 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:32:50,441 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38374.77 MB 2025-02-14 15:32:50,441 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38474.68 MB 2025-02-14 15:32:50,441 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 99.91 MB 2025-02-14 15:32:50,441 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44931.48 MB 2025-02-14 15:32:50,441 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44931.48 MB 2025-02-14 15:32:50,441 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:32:50,441 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39074.17 MB 2025-02-14 15:32:50,458 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8117, cut from 8119 2025-02-14 15:32:50,459 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 15:32:50,465 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 15:32:50,465 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 15:32:50,465 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:32:50,465 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:32:50,465 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28045.95 MB 2025-02-14 15:32:50,465 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32217.35 MB 2025-02-14 15:32:50,465 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4171.40 MB 2025-02-14 15:32:50,465 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44931.48 MB 2025-02-14 15:32:50,465 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53276.05 MB 2025-02-14 15:32:50,465 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8344.57 MB 2025-02-14 15:32:50,465 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36388.58 MB 2025-02-14 15:32:50,616 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7909] 2025-02-14 15:32:50,618 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:32:50,618 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:32:50,618 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:32:50,618 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 15:32:50,623 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 15:32:50,624 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:32:50,624 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 15:32:50,624 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 15:32:50,625 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:32:50,625 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:32:50,625 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:32:50,626 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:32:50,631 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 15:32:50,632 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:32:50,632 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:32:50,632 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:32:50,632 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:32:50,632 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 15:32:50,633 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:32:50,633 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:32:50,633 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:32:50,633 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 15:32:50,633 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 15:32:50,634 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:32:50,634 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:32:50,638 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:32:50,638 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:32:50,638 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:32:50,638 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:32:50,639 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:32:50,639 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:32:50,641 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:32:50,642 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:33:45,567 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:33:45,568 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:33:45,576 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 15:33:45,579 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:33:45,579 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1214, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 15:33:45,581 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:33:45,581 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1214, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 15:34:04,271 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 15:34:04,272 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 15:34:04,272 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.68 seconds 2025-02-14 15:34:04,272 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:34:04,272 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31656.11 MB 2025-02-14 15:34:04,272 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35953.17 MB 2025-02-14 15:34:04,272 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4297.06 MB 2025-02-14 15:34:04,272 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53276.05 MB 2025-02-14 15:34:04,272 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50551.85 MB 2025-02-14 15:34:04,272 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2724.20 MB 2025-02-14 15:34:04,272 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44751.35 MB 2025-02-14 15:34:04,347 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 15:34:04,347 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 15:34:04,348 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 15:34:04,348 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:34:04,348 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35953.17 MB 2025-02-14 15:34:04,348 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32317.11 MB 2025-02-14 15:34:04,348 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3636.06 MB 2025-02-14 15:34:04,348 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50551.85 MB 2025-02-14 15:34:04,348 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56998.49 MB 2025-02-14 15:34:04,348 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6446.65 MB 2025-02-14 15:34:04,348 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48825.77 MB 2025-02-14 15:34:06,260 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 15:34:06,260 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 15:34:06,260 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 15:34:06,260 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:34:06,260 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32317.11 MB 2025-02-14 15:34:06,260 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32847.95 MB 2025-02-14 15:34:06,260 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 15:34:06,260 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56998.49 MB 2025-02-14 15:34:06,260 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46254.78 MB 2025-02-14 15:34:06,260 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10743.71 MB 2025-02-14 15:34:06,260 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36827.29 MB 2025-02-14 15:34:06,274 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 15:34:06,274 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 15:34:06,274 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:34:06,274 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:34:06,274 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32847.95 MB 2025-02-14 15:34:06,274 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34737.49 MB 2025-02-14 15:34:06,274 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 15:34:06,274 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46254.78 MB 2025-02-14 15:34:06,274 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46254.78 MB 2025-02-14 15:34:06,274 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:34:06,274 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36154.92 MB 2025-02-14 15:34:06,486 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 15:34:06,486 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 15:34:06,486 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 15:34:06,486 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:34:06,486 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34737.49 MB 2025-02-14 15:34:06,486 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36979.34 MB 2025-02-14 15:34:06,486 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 15:34:06,486 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46254.78 MB 2025-02-14 15:34:06,486 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46254.78 MB 2025-02-14 15:34:06,486 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:34:06,486 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42523.62 MB 2025-02-14 15:34:06,487 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 15:34:06,487 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 15:34:06,487 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 15:34:06,487 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:34:06,487 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32847.95 MB 2025-02-14 15:34:06,487 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36979.34 MB 2025-02-14 15:34:06,487 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 15:34:06,487 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46254.78 MB 2025-02-14 15:34:06,487 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46254.78 MB 2025-02-14 15:34:06,487 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:34:06,487 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42523.62 MB 2025-02-14 15:34:06,658 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 15:34:06,658 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 15:34:06,658 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 15:34:06,658 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:34:06,658 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37687.13 MB 2025-02-14 15:34:06,658 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38454.13 MB 2025-02-14 15:34:06,658 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 15:34:06,658 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46254.78 MB 2025-02-14 15:34:06,658 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46670.02 MB 2025-02-14 15:34:06,658 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 15:34:06,658 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39161.92 MB 2025-02-14 15:34:06,675 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 15:34:06,676 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 15:34:06,676 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:34:06,676 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:34:06,676 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38867.02 MB 2025-02-14 15:34:06,676 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39073.85 MB 2025-02-14 15:34:06,676 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.83 MB 2025-02-14 15:34:06,676 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46670.02 MB 2025-02-14 15:34:06,676 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46670.02 MB 2025-02-14 15:34:06,676 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:34:06,676 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39295.77 MB 2025-02-14 15:34:06,677 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 15:34:06,677 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 15:34:06,677 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.09 seconds 2025-02-14 15:34:06,677 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:34:06,677 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27426.43 MB 2025-02-14 15:34:06,677 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39274.80 MB 2025-02-14 15:34:06,677 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11848.37 MB 2025-02-14 15:34:06,677 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53276.05 MB 2025-02-14 15:34:06,677 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46670.02 MB 2025-02-14 15:34:06,677 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6606.03 MB 2025-02-14 15:34:06,677 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39295.77 MB 2025-02-14 15:34:06,942 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 15:34:06,942 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 15:34:06,942 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 15:34:06,942 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:34:06,942 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39274.80 MB 2025-02-14 15:34:06,942 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39375.21 MB 2025-02-14 15:34:06,942 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.41 MB 2025-02-14 15:34:06,942 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46670.02 MB 2025-02-14 15:34:06,942 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46670.02 MB 2025-02-14 15:34:06,942 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:34:06,942 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39977.64 MB 2025-02-14 15:34:06,960 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8157, cut from 8159 2025-02-14 15:34:06,960 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 15:34:06,966 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 15:34:06,966 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 15:34:06,966 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:34:06,966 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:34:06,966 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28689.23 MB 2025-02-14 15:34:06,966 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32881.15 MB 2025-02-14 15:34:06,966 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4191.92 MB 2025-02-14 15:34:06,966 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46670.02 MB 2025-02-14 15:34:06,966 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50862.23 MB 2025-02-14 15:34:06,966 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4192.21 MB 2025-02-14 15:34:06,966 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37073.36 MB 2025-02-14 15:34:07,128 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7949] 2025-02-14 15:34:07,129 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:34:07,129 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:34:07,130 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:34:07,130 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 15:34:07,135 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 15:34:07,136 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:34:07,136 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 15:34:07,136 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 15:34:07,137 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:34:07,137 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:34:07,138 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:34:07,138 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:34:07,144 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 15:34:07,144 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:34:07,144 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:34:07,145 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:34:07,145 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:34:07,145 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 15:34:07,145 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:34:07,145 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:34:07,146 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:34:07,146 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 15:34:07,146 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 15:34:07,146 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:34:07,146 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:34:07,152 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:34:07,152 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:34:07,154 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:34:07,154 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:34:07,156 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:34:07,156 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:34:07,159 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:34:07,160 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:35:00,487 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:35:00,488 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:35:00,496 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 15:35:00,499 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:35:00,499 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1215, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 15:35:00,501 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:35:00,501 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1215, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 15:35:19,242 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 15:35:19,242 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 15:35:19,242 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.73 seconds 2025-02-14 15:35:19,242 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:35:19,242 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31784.83 MB 2025-02-14 15:35:19,242 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36084.65 MB 2025-02-14 15:35:19,242 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4299.82 MB 2025-02-14 15:35:19,242 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50862.23 MB 2025-02-14 15:35:19,242 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46357.54 MB 2025-02-14 15:35:19,242 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4504.68 MB 2025-02-14 15:35:19,242 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44962.81 MB 2025-02-14 15:35:19,320 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 15:35:19,320 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 15:35:19,320 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 15:35:19,320 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:35:19,320 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36084.65 MB 2025-02-14 15:35:19,320 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32444.07 MB 2025-02-14 15:35:19,320 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3640.58 MB 2025-02-14 15:35:19,320 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46357.54 MB 2025-02-14 15:35:19,320 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54857.30 MB 2025-02-14 15:35:19,320 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8499.76 MB 2025-02-14 15:35:19,320 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48918.15 MB 2025-02-14 15:35:21,233 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 15:35:21,233 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 15:35:21,233 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 15:35:21,233 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:35:21,233 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32444.07 MB 2025-02-14 15:35:21,233 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32974.91 MB 2025-02-14 15:35:21,233 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 15:35:21,233 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54857.30 MB 2025-02-14 15:35:21,233 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42056.29 MB 2025-02-14 15:35:21,233 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12801.02 MB 2025-02-14 15:35:21,233 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36954.24 MB 2025-02-14 15:35:21,246 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 15:35:21,246 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 15:35:21,246 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:35:21,246 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:35:21,246 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32974.91 MB 2025-02-14 15:35:21,246 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34864.44 MB 2025-02-14 15:35:21,246 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 15:35:21,247 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42056.29 MB 2025-02-14 15:35:21,247 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42056.29 MB 2025-02-14 15:35:21,247 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:35:21,247 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36281.87 MB 2025-02-14 15:35:21,468 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 15:35:21,468 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 15:35:21,468 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 15:35:21,468 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:35:21,468 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34864.44 MB 2025-02-14 15:35:21,468 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37106.30 MB 2025-02-14 15:35:21,468 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 15:35:21,468 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42056.29 MB 2025-02-14 15:35:21,468 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45831.16 MB 2025-02-14 15:35:21,468 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 15:35:21,468 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42650.58 MB 2025-02-14 15:35:21,468 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 15:35:21,468 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 15:35:21,468 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 15:35:21,469 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:35:21,469 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32974.91 MB 2025-02-14 15:35:21,469 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37106.30 MB 2025-02-14 15:35:21,469 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 15:35:21,469 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42056.29 MB 2025-02-14 15:35:21,469 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45831.16 MB 2025-02-14 15:35:21,469 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 15:35:21,469 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42650.58 MB 2025-02-14 15:35:21,637 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 15:35:21,637 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 15:35:21,637 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 15:35:21,637 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:35:21,637 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37814.09 MB 2025-02-14 15:35:21,637 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38581.09 MB 2025-02-14 15:35:21,637 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 15:35:21,637 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45831.16 MB 2025-02-14 15:35:21,637 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46246.40 MB 2025-02-14 15:35:21,637 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 15:35:21,637 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39288.88 MB 2025-02-14 15:35:21,655 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 15:35:21,655 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 15:35:21,655 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:35:21,655 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:35:21,655 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38993.98 MB 2025-02-14 15:35:21,655 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39199.79 MB 2025-02-14 15:35:21,655 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.81 MB 2025-02-14 15:35:21,655 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46246.40 MB 2025-02-14 15:35:21,655 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46246.40 MB 2025-02-14 15:35:21,655 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:35:21,655 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39423.91 MB 2025-02-14 15:35:21,656 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 15:35:21,656 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 15:35:21,656 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.15 seconds 2025-02-14 15:35:21,656 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:35:21,656 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27551.67 MB 2025-02-14 15:35:21,656 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39400.15 MB 2025-02-14 15:35:21,656 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11848.47 MB 2025-02-14 15:35:21,656 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50862.23 MB 2025-02-14 15:35:21,656 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46246.40 MB 2025-02-14 15:35:21,656 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4615.83 MB 2025-02-14 15:35:21,656 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39423.91 MB 2025-02-14 15:35:21,919 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 15:35:21,919 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 15:35:21,919 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 15:35:21,919 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:35:21,919 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39400.15 MB 2025-02-14 15:35:21,919 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39500.26 MB 2025-02-14 15:35:21,919 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.11 MB 2025-02-14 15:35:21,919 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46246.40 MB 2025-02-14 15:35:21,919 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46246.40 MB 2025-02-14 15:35:21,919 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:35:21,919 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40100.92 MB 2025-02-14 15:35:21,937 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8133, cut from 8135 2025-02-14 15:35:21,937 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-14 15:35:21,944 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 15:35:21,944 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 15:35:21,944 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:35:21,944 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:35:21,944 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28813.88 MB 2025-02-14 15:35:21,944 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32993.51 MB 2025-02-14 15:35:21,944 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4179.62 MB 2025-02-14 15:35:21,944 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46246.40 MB 2025-02-14 15:35:21,944 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54605.64 MB 2025-02-14 15:35:21,944 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8359.25 MB 2025-02-14 15:35:21,944 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37173.13 MB 2025-02-14 15:35:22,107 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7925] 2025-02-14 15:35:22,109 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:35:22,109 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:35:22,110 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:35:22,110 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 15:35:22,114 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 15:35:22,115 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:35:22,115 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 15:35:22,116 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-14 15:35:22,116 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:35:22,116 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:35:22,117 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:35:22,117 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:35:22,123 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 15:35:22,123 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:35:22,123 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:35:22,124 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:35:22,124 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:35:22,124 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 15:35:22,124 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:35:22,124 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:35:22,125 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:35:22,125 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 15:35:22,125 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 15:35:22,125 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:35:22,125 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:35:22,132 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:35:22,132 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:35:22,134 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:35:22,134 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:35:22,136 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:35:22,136 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:35:22,139 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:35:22,140 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:36:21,358 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:36:21,358 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:36:21,363 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 15:36:21,364 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:36:21,364 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1174, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 15:36:21,365 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:36:21,365 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1174, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 15:36:39,427 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 15:36:39,427 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 15:36:39,427 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.05 seconds 2025-02-14 15:36:39,427 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:36:39,427 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31620.91 MB 2025-02-14 15:36:39,427 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35775.63 MB 2025-02-14 15:36:39,427 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4154.72 MB 2025-02-14 15:36:39,427 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54605.64 MB 2025-02-14 15:36:39,427 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42020.63 MB 2025-02-14 15:36:39,427 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12585.01 MB 2025-02-14 15:36:39,427 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44716.16 MB 2025-02-14 15:36:39,533 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 15:36:39,533 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 15:36:39,533 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 15:36:39,533 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:36:39,533 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35775.63 MB 2025-02-14 15:36:39,533 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32352.70 MB 2025-02-14 15:36:39,533 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3422.94 MB 2025-02-14 15:36:39,533 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42020.63 MB 2025-02-14 15:36:39,533 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54127.49 MB 2025-02-14 15:36:39,533 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 12106.86 MB 2025-02-14 15:36:39,533 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48233.38 MB 2025-02-14 15:36:41,461 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 15:36:41,461 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 15:36:41,461 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 15:36:41,461 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:36:41,461 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32352.70 MB 2025-02-14 15:36:41,461 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32883.54 MB 2025-02-14 15:36:41,461 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 15:36:41,461 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54127.49 MB 2025-02-14 15:36:41,461 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37180.41 MB 2025-02-14 15:36:41,461 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16947.09 MB 2025-02-14 15:36:41,461 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36863.91 MB 2025-02-14 15:36:41,475 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 15:36:41,475 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 15:36:41,475 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:36:41,475 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:36:41,475 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32883.54 MB 2025-02-14 15:36:41,475 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34772.89 MB 2025-02-14 15:36:41,475 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.35 MB 2025-02-14 15:36:41,475 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37180.41 MB 2025-02-14 15:36:41,475 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39067.84 MB 2025-02-14 15:36:41,475 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 15:36:41,475 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36190.32 MB 2025-02-14 15:36:41,689 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 15:36:41,689 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 15:36:41,689 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 15:36:41,689 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:36:41,689 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34772.89 MB 2025-02-14 15:36:41,689 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37014.75 MB 2025-02-14 15:36:41,689 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 15:36:41,689 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39067.84 MB 2025-02-14 15:36:41,689 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45202.01 MB 2025-02-14 15:36:41,689 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 15:36:41,689 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42559.03 MB 2025-02-14 15:36:41,690 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 15:36:41,690 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 15:36:41,690 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 15:36:41,690 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:36:41,690 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32883.54 MB 2025-02-14 15:36:41,690 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37014.75 MB 2025-02-14 15:36:41,690 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.21 MB 2025-02-14 15:36:41,690 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37180.41 MB 2025-02-14 15:36:41,690 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45202.01 MB 2025-02-14 15:36:41,690 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-14 15:36:41,690 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42559.03 MB 2025-02-14 15:36:41,855 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 15:36:41,855 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 15:36:41,855 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 15:36:41,855 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:36:41,855 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37722.54 MB 2025-02-14 15:36:41,855 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38489.54 MB 2025-02-14 15:36:41,855 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 15:36:41,855 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45202.01 MB 2025-02-14 15:36:41,855 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45617.25 MB 2025-02-14 15:36:41,855 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 15:36:41,855 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39197.33 MB 2025-02-14 15:36:41,873 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 15:36:41,873 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 15:36:41,873 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:36:41,873 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:36:41,873 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38902.43 MB 2025-02-14 15:36:41,873 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39108.37 MB 2025-02-14 15:36:41,873 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.95 MB 2025-02-14 15:36:41,873 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45617.25 MB 2025-02-14 15:36:41,873 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45617.25 MB 2025-02-14 15:36:41,873 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:36:41,873 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39328.23 MB 2025-02-14 15:36:41,874 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 15:36:41,874 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 15:36:41,874 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.51 seconds 2025-02-14 15:36:41,874 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:36:41,874 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27530.60 MB 2025-02-14 15:36:41,874 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39309.40 MB 2025-02-14 15:36:41,874 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11778.79 MB 2025-02-14 15:36:41,874 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54605.64 MB 2025-02-14 15:36:41,874 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45617.25 MB 2025-02-14 15:36:41,874 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8988.39 MB 2025-02-14 15:36:41,874 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39328.23 MB 2025-02-14 15:36:42,139 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 15:36:42,139 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 15:36:42,139 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 15:36:42,139 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:36:42,139 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39309.40 MB 2025-02-14 15:36:42,139 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39409.84 MB 2025-02-14 15:36:42,139 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.44 MB 2025-02-14 15:36:42,139 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45617.25 MB 2025-02-14 15:36:42,139 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45617.25 MB 2025-02-14 15:36:42,139 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:36:42,139 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40012.49 MB 2025-02-14 15:36:42,157 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8160, cut from 8162 2025-02-14 15:36:42,157 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 15:36:42,163 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 15:36:42,163 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 15:36:42,163 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:36:42,163 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:36:42,163 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28793.47 MB 2025-02-14 15:36:42,163 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32987.78 MB 2025-02-14 15:36:42,163 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4194.31 MB 2025-02-14 15:36:42,163 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45617.25 MB 2025-02-14 15:36:42,163 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56103.01 MB 2025-02-14 15:36:42,163 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10485.76 MB 2025-02-14 15:36:42,163 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37180.72 MB 2025-02-14 15:36:42,325 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7952] 2025-02-14 15:36:42,326 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:36:42,326 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:36:42,327 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:36:42,327 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 15:36:42,332 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 15:36:42,333 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:36:42,333 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 15:36:42,333 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 15:36:42,334 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:36:42,334 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:36:42,334 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:36:42,334 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:36:42,340 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 15:36:42,341 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:36:42,341 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:36:42,341 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:36:42,341 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:36:42,341 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 15:36:42,342 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:36:42,342 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:36:42,342 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:36:42,342 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 15:36:42,342 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 15:36:42,343 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:36:42,343 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:36:42,348 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:36:42,348 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:36:42,349 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:36:42,349 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:36:42,351 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:36:42,351 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:36:42,353 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:36:42,353 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:37:40,001 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:37:40,001 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:37:40,006 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 15:37:40,008 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:37:40,008 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1265, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 15:37:40,008 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:37:40,009 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1265, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 15:37:59,430 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 15:37:59,430 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 15:37:59,430 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.41 seconds 2025-02-14 15:37:59,430 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:37:59,430 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32376.43 MB 2025-02-14 15:37:59,431 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36853.85 MB 2025-02-14 15:37:59,431 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4477.42 MB 2025-02-14 15:37:59,431 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58273.56 MB 2025-02-14 15:37:59,431 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48702.16 MB 2025-02-14 15:37:59,431 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9571.40 MB 2025-02-14 15:37:59,431 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45698.84 MB 2025-02-14 15:37:59,507 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 15:37:59,507 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 15:37:59,507 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 15:37:59,507 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:37:59,507 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36853.85 MB 2025-02-14 15:37:59,507 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32947.19 MB 2025-02-14 15:37:59,507 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3906.66 MB 2025-02-14 15:37:59,507 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48702.16 MB 2025-02-14 15:37:59,507 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57556.34 MB 2025-02-14 15:37:59,507 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8854.18 MB 2025-02-14 15:37:59,507 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50166.18 MB 2025-02-14 15:38:01,425 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 15:38:01,425 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 15:38:01,425 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 15:38:01,425 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:38:01,425 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32947.19 MB 2025-02-14 15:38:01,425 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33478.03 MB 2025-02-14 15:38:01,425 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 15:38:01,425 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57556.34 MB 2025-02-14 15:38:01,425 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40051.41 MB 2025-02-14 15:38:01,425 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17504.93 MB 2025-02-14 15:38:01,425 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37457.37 MB 2025-02-14 15:38:01,438 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 15:38:01,438 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 15:38:01,438 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:38:01,438 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:38:01,438 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33478.03 MB 2025-02-14 15:38:01,438 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35367.57 MB 2025-02-14 15:38:01,438 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 15:38:01,438 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40051.41 MB 2025-02-14 15:38:01,438 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40051.41 MB 2025-02-14 15:38:01,438 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:38:01,438 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36785.00 MB 2025-02-14 15:38:01,652 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 15:38:01,652 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 15:38:01,653 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 15:38:01,653 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:38:01,653 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35367.57 MB 2025-02-14 15:38:01,653 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37609.42 MB 2025-02-14 15:38:01,653 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 15:38:01,653 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40051.41 MB 2025-02-14 15:38:01,653 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45713.72 MB 2025-02-14 15:38:01,653 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 15:38:01,653 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43153.70 MB 2025-02-14 15:38:01,653 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 15:38:01,653 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 15:38:01,653 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 15:38:01,653 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:38:01,653 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33478.03 MB 2025-02-14 15:38:01,653 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37609.42 MB 2025-02-14 15:38:01,653 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 15:38:01,653 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40051.41 MB 2025-02-14 15:38:01,653 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45713.72 MB 2025-02-14 15:38:01,653 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 15:38:01,653 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43153.70 MB 2025-02-14 15:38:01,818 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 15:38:01,818 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 15:38:01,818 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 15:38:01,818 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:38:01,818 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38317.21 MB 2025-02-14 15:38:01,818 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39084.21 MB 2025-02-14 15:38:01,818 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 15:38:01,818 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45713.72 MB 2025-02-14 15:38:01,818 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46128.96 MB 2025-02-14 15:38:01,818 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 15:38:01,818 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39792.00 MB 2025-02-14 15:38:01,836 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 15:38:01,836 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 15:38:01,836 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:38:01,836 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:38:01,836 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39497.10 MB 2025-02-14 15:38:01,836 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39703.57 MB 2025-02-14 15:38:01,836 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.47 MB 2025-02-14 15:38:01,836 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46128.96 MB 2025-02-14 15:38:01,836 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46128.96 MB 2025-02-14 15:38:01,836 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:38:01,836 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39924.90 MB 2025-02-14 15:38:01,837 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 15:38:01,837 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 15:38:01,837 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.83 seconds 2025-02-14 15:38:01,837 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:38:01,837 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27969.07 MB 2025-02-14 15:38:01,837 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39904.64 MB 2025-02-14 15:38:01,837 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11935.57 MB 2025-02-14 15:38:01,837 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58273.56 MB 2025-02-14 15:38:01,837 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46128.96 MB 2025-02-14 15:38:01,837 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12144.61 MB 2025-02-14 15:38:01,837 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39924.90 MB 2025-02-14 15:38:02,102 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 15:38:02,102 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 15:38:02,102 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 15:38:02,102 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:38:02,102 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39904.64 MB 2025-02-14 15:38:02,102 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40005.11 MB 2025-02-14 15:38:02,102 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.47 MB 2025-02-14 15:38:02,102 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46128.96 MB 2025-02-14 15:38:02,102 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46128.96 MB 2025-02-14 15:38:02,102 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:38:02,102 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40607.91 MB 2025-02-14 15:38:02,120 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 15:38:02,120 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 15:38:02,126 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 15:38:02,126 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 15:38:02,126 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:38:02,126 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:38:02,126 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29231.99 MB 2025-02-14 15:38:02,126 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33426.47 MB 2025-02-14 15:38:02,126 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4194.49 MB 2025-02-14 15:38:02,126 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46128.96 MB 2025-02-14 15:38:02,126 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54519.66 MB 2025-02-14 15:38:02,126 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 15:38:02,126 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37620.78 MB 2025-02-14 15:38:02,288 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 15:38:02,289 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:38:02,289 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:38:02,290 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:38:02,290 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 15:38:02,295 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 15:38:02,296 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:38:02,296 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 15:38:02,296 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 15:38:02,297 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:38:02,297 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:38:02,297 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:38:02,297 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:38:02,303 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 15:38:02,303 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:38:02,303 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:38:02,304 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:38:02,304 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:38:02,304 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 15:38:02,304 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:38:02,304 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:38:02,305 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:38:02,305 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 15:38:02,305 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 15:38:02,306 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:38:02,306 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:38:02,311 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:38:02,311 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:38:02,312 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:38:02,312 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:38:02,313 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:38:02,313 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:38:02,317 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:38:02,317 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:38:08,158 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:38:08,158 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:38:08,163 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 15:38:08,164 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:38:08,164 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1245, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 15:38:08,165 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:38:08,165 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1245, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 15:38:27,482 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 15:38:27,482 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 15:38:27,482 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.31 seconds 2025-02-14 15:38:27,482 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:38:27,482 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32358.29 MB 2025-02-14 15:38:27,482 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36764.41 MB 2025-02-14 15:38:27,482 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4406.12 MB 2025-02-14 15:38:27,482 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54519.66 MB 2025-02-14 15:38:27,482 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52848.23 MB 2025-02-14 15:38:27,482 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1671.43 MB 2025-02-14 15:38:27,482 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45680.03 MB 2025-02-14 15:38:27,555 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 15:38:27,555 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 15:38:27,555 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 15:38:27,555 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:38:27,555 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36764.41 MB 2025-02-14 15:38:27,555 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32964.44 MB 2025-02-14 15:38:27,555 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3799.96 MB 2025-02-14 15:38:27,555 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52848.23 MB 2025-02-14 15:38:27,555 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 59389.25 MB 2025-02-14 15:38:27,555 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6541.02 MB 2025-02-14 15:38:27,556 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49960.45 MB 2025-02-14 15:38:29,490 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 15:38:29,490 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 15:38:29,490 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 15:38:29,490 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:38:29,490 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32964.44 MB 2025-02-14 15:38:29,490 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33495.29 MB 2025-02-14 15:38:29,490 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 15:38:29,490 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59389.25 MB 2025-02-14 15:38:29,490 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48442.11 MB 2025-02-14 15:38:29,490 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10947.13 MB 2025-02-14 15:38:29,490 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37474.62 MB 2025-02-14 15:38:29,504 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 15:38:29,504 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 15:38:29,504 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:38:29,504 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:38:29,504 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33495.29 MB 2025-02-14 15:38:29,504 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35384.64 MB 2025-02-14 15:38:29,504 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.35 MB 2025-02-14 15:38:29,504 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48442.11 MB 2025-02-14 15:38:29,504 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48442.11 MB 2025-02-14 15:38:29,504 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:38:29,504 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36802.07 MB 2025-02-14 15:38:29,723 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 15:38:29,723 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 15:38:29,723 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 15:38:29,723 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:38:29,723 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35384.64 MB 2025-02-14 15:38:29,723 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37626.50 MB 2025-02-14 15:38:29,723 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 15:38:29,723 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48442.11 MB 2025-02-14 15:38:29,723 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48442.11 MB 2025-02-14 15:38:29,723 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:38:29,723 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43170.78 MB 2025-02-14 15:38:29,723 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 15:38:29,723 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 15:38:29,724 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 15:38:29,724 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:38:29,724 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33495.29 MB 2025-02-14 15:38:29,724 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37626.50 MB 2025-02-14 15:38:29,724 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.21 MB 2025-02-14 15:38:29,724 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48442.11 MB 2025-02-14 15:38:29,724 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48442.11 MB 2025-02-14 15:38:29,724 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:38:29,724 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43170.78 MB 2025-02-14 15:38:29,887 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 15:38:29,887 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 15:38:29,887 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 15:38:29,887 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:38:29,887 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38334.28 MB 2025-02-14 15:38:29,887 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39101.29 MB 2025-02-14 15:38:29,887 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 15:38:29,887 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48442.11 MB 2025-02-14 15:38:29,887 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48857.35 MB 2025-02-14 15:38:29,887 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 15:38:29,887 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39809.08 MB 2025-02-14 15:38:29,904 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 15:38:29,904 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 15:38:29,904 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:38:29,904 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:38:29,904 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39514.18 MB 2025-02-14 15:38:29,904 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39719.91 MB 2025-02-14 15:38:29,904 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.74 MB 2025-02-14 15:38:29,904 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48857.35 MB 2025-02-14 15:38:29,904 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48857.35 MB 2025-02-14 15:38:29,904 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:38:29,904 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39944.49 MB 2025-02-14 15:38:29,905 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 15:38:29,905 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 15:38:29,905 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.74 seconds 2025-02-14 15:38:29,905 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:38:29,905 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28020.61 MB 2025-02-14 15:38:29,905 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39920.35 MB 2025-02-14 15:38:29,905 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11899.73 MB 2025-02-14 15:38:29,905 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54519.66 MB 2025-02-14 15:38:29,905 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48857.35 MB 2025-02-14 15:38:29,905 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5662.31 MB 2025-02-14 15:38:29,905 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39944.49 MB 2025-02-14 15:38:30,173 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 15:38:30,173 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 15:38:30,173 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 15:38:30,173 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:38:30,173 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39920.35 MB 2025-02-14 15:38:30,173 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40020.49 MB 2025-02-14 15:38:30,173 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.15 MB 2025-02-14 15:38:30,173 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48857.35 MB 2025-02-14 15:38:30,173 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48857.35 MB 2025-02-14 15:38:30,173 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:38:30,173 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40621.38 MB 2025-02-14 15:38:30,191 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8136, cut from 8138 2025-02-14 15:38:30,191 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 15:38:30,197 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 15:38:30,197 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 15:38:30,197 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:38:30,197 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:38:30,197 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29282.89 MB 2025-02-14 15:38:30,197 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33464.04 MB 2025-02-14 15:38:30,197 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4181.15 MB 2025-02-14 15:38:30,198 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48857.35 MB 2025-02-14 15:38:30,198 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53039.07 MB 2025-02-14 15:38:30,198 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4181.72 MB 2025-02-14 15:38:30,198 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37644.67 MB 2025-02-14 15:38:30,355 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7928] 2025-02-14 15:38:30,356 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:38:30,356 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:38:30,357 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:38:30,357 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 15:38:30,362 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 15:38:30,363 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:38:30,363 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 15:38:30,363 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 15:38:30,364 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:38:30,364 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:38:30,364 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:38:30,364 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:38:30,370 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 15:38:30,371 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:38:30,371 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:38:30,371 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:38:30,371 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:38:30,371 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 15:38:30,372 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:38:30,372 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:38:30,372 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:38:30,372 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 15:38:30,372 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 15:38:30,373 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:38:30,373 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:38:30,376 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:38:30,376 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:38:30,377 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:38:30,377 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:38:30,378 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:38:30,378 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:38:30,380 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:38:30,380 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:39:30,075 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:39:30,075 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:39:30,080 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 15:39:30,081 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:39:30,081 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 136, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 15:39:30,082 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:39:30,082 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 136, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 15:39:32,229 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 15:39:32,229 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 15:39:32,229 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.14 seconds 2025-02-14 15:39:32,229 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:39:32,229 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24752.57 MB 2025-02-14 15:39:32,229 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25233.86 MB 2025-02-14 15:39:32,229 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 481.30 MB 2025-02-14 15:39:32,229 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53039.07 MB 2025-02-14 15:39:32,229 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29492.25 MB 2025-02-14 15:39:32,229 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23546.82 MB 2025-02-14 15:39:32,229 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34224.74 MB 2025-02-14 15:39:32,240 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 15:39:32,240 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 15:39:32,240 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:39:32,240 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:39:32,240 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25233.86 MB 2025-02-14 15:39:32,240 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25424.91 MB 2025-02-14 15:39:32,240 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 191.05 MB 2025-02-14 15:39:32,240 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29492.25 MB 2025-02-14 15:39:32,240 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29492.25 MB 2025-02-14 15:39:32,240 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:39:32,240 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27088.22 MB 2025-02-14 15:39:32,879 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 15:39:32,879 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 15:39:32,879 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.64 seconds 2025-02-14 15:39:32,879 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:39:32,879 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25424.91 MB 2025-02-14 15:39:32,879 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25597.43 MB 2025-02-14 15:39:32,879 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 172.52 MB 2025-02-14 15:39:32,879 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29492.25 MB 2025-02-14 15:39:32,879 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29492.25 MB 2025-02-14 15:39:32,879 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:39:32,879 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29596.39 MB 2025-02-14 15:39:32,887 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 15:39:32,887 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 15:39:32,887 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 15:39:32,887 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:39:32,887 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25597.37 MB 2025-02-14 15:39:32,887 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26211.32 MB 2025-02-14 15:39:32,887 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 613.95 MB 2025-02-14 15:39:32,887 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29492.25 MB 2025-02-14 15:39:32,887 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29492.25 MB 2025-02-14 15:39:32,887 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:39:32,887 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26671.99 MB 2025-02-14 15:39:32,963 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 15:39:32,963 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 15:39:32,963 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 15:39:32,963 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:39:32,963 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26211.32 MB 2025-02-14 15:39:32,963 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26939.97 MB 2025-02-14 15:39:32,963 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 728.65 MB 2025-02-14 15:39:32,963 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29492.25 MB 2025-02-14 15:39:32,963 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30417.09 MB 2025-02-14 15:39:32,963 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 924.84 MB 2025-02-14 15:39:32,963 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28741.81 MB 2025-02-14 15:39:32,963 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 15:39:32,963 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 15:39:32,963 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 15:39:32,963 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:39:32,963 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25597.37 MB 2025-02-14 15:39:32,963 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26939.97 MB 2025-02-14 15:39:32,963 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1342.60 MB 2025-02-14 15:39:32,963 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29492.25 MB 2025-02-14 15:39:32,964 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30417.09 MB 2025-02-14 15:39:32,964 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 924.84 MB 2025-02-14 15:39:32,964 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28741.81 MB 2025-02-14 15:39:33,025 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 15:39:33,025 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 15:39:33,025 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 15:39:33,026 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:39:33,026 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27170.00 MB 2025-02-14 15:39:33,026 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27419.27 MB 2025-02-14 15:39:33,026 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 249.28 MB 2025-02-14 15:39:33,026 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30417.09 MB 2025-02-14 15:39:33,026 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30549.21 MB 2025-02-14 15:39:33,026 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 132.12 MB 2025-02-14 15:39:33,026 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27663.13 MB 2025-02-14 15:39:33,033 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 15:39:33,033 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 15:39:33,033 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:39:33,033 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:39:33,033 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27553.47 MB 2025-02-14 15:39:33,033 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27755.47 MB 2025-02-14 15:39:33,034 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 201.99 MB 2025-02-14 15:39:33,034 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30549.21 MB 2025-02-14 15:39:33,034 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30553.41 MB 2025-02-14 15:39:33,034 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-14 15:39:33,034 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27755.47 MB 2025-02-14 15:39:33,035 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 15:39:33,035 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 15:39:33,035 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.95 seconds 2025-02-14 15:39:33,035 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:39:33,035 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24278.73 MB 2025-02-14 15:39:33,035 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27956.24 MB 2025-02-14 15:39:33,035 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3677.51 MB 2025-02-14 15:39:33,035 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53039.07 MB 2025-02-14 15:39:33,035 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30553.41 MB 2025-02-14 15:39:33,035 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22485.66 MB 2025-02-14 15:39:33,035 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27956.24 MB 2025-02-14 15:39:33,298 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 15:39:33,298 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 15:39:33,298 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 15:39:33,298 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:39:33,298 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27956.24 MB 2025-02-14 15:39:33,298 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28056.56 MB 2025-02-14 15:39:33,298 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.32 MB 2025-02-14 15:39:33,298 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30553.41 MB 2025-02-14 15:39:33,298 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30553.41 MB 2025-02-14 15:39:33,298 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:39:33,298 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28658.78 MB 2025-02-14 15:39:33,317 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8150, cut from 8152 2025-02-14 15:39:33,317 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2.'] 2025-02-14 15:39:33,323 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 15:39:33,323 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 15:39:33,323 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:39:33,323 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:39:33,323 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24824.57 MB 2025-02-14 15:39:33,323 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29012.89 MB 2025-02-14 15:39:33,323 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4188.33 MB 2025-02-14 15:39:33,323 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30553.41 MB 2025-02-14 15:39:33,323 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41026.58 MB 2025-02-14 15:39:33,323 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10473.18 MB 2025-02-14 15:39:33,323 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33200.91 MB 2025-02-14 15:39:33,475 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7942] 2025-02-14 15:39:33,476 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:39:33,476 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:39:33,477 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:39:33,477 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 15:39:33,481 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 15:39:33,482 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:39:33,482 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 15:39:33,482 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2.'] 2025-02-14 15:39:33,483 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:39:33,483 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:39:33,484 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:39:33,484 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:39:33,489 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 15:39:33,490 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:39:33,490 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:39:33,490 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:39:33,490 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:39:33,490 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 15:39:33,491 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:39:33,491 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:39:33,491 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:39:33,491 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 15:39:33,491 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 15:39:33,492 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:39:33,492 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:39:33,496 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:39:33,496 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:39:33,497 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:39:33,497 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:39:33,498 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:39:33,498 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:39:33,501 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:39:33,501 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:39:42,077 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:39:42,078 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:39:42,082 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 15:39:42,084 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:39:42,084 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1330, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 15:39:42,085 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:39:42,085 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1330, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 15:40:02,561 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 15:40:02,561 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 15:40:02,561 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.47 seconds 2025-02-14 15:40:02,561 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:40:02,561 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33193.83 MB 2025-02-14 15:40:02,561 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37900.62 MB 2025-02-14 15:40:02,561 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4706.80 MB 2025-02-14 15:40:02,561 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45589.99 MB 2025-02-14 15:40:02,561 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47141.88 MB 2025-02-14 15:40:02,561 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1551.89 MB 2025-02-14 15:40:02,561 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46742.06 MB 2025-02-14 15:40:02,639 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 15:40:02,640 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 15:40:02,640 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 15:40:02,640 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:40:02,640 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37900.62 MB 2025-02-14 15:40:02,640 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33649.58 MB 2025-02-14 15:40:02,640 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4251.05 MB 2025-02-14 15:40:02,640 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47141.88 MB 2025-02-14 15:40:02,640 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56818.14 MB 2025-02-14 15:40:02,640 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9676.26 MB 2025-02-14 15:40:02,640 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51852.13 MB 2025-02-14 15:40:04,560 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 15:40:04,560 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 15:40:04,560 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 15:40:04,560 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:40:04,560 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33649.58 MB 2025-02-14 15:40:04,560 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34180.42 MB 2025-02-14 15:40:04,560 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 15:40:04,560 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56818.14 MB 2025-02-14 15:40:04,560 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42433.77 MB 2025-02-14 15:40:04,560 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14384.37 MB 2025-02-14 15:40:04,560 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38159.75 MB 2025-02-14 15:40:04,574 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 15:40:04,574 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 15:40:04,574 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:40:04,574 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:40:04,574 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34180.42 MB 2025-02-14 15:40:04,574 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36069.95 MB 2025-02-14 15:40:04,574 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 15:40:04,574 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42433.77 MB 2025-02-14 15:40:04,574 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42433.77 MB 2025-02-14 15:40:04,574 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:40:04,574 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37487.38 MB 2025-02-14 15:40:04,782 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 15:40:04,782 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 15:40:04,782 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 15:40:04,782 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:40:04,782 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36069.95 MB 2025-02-14 15:40:04,782 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38311.81 MB 2025-02-14 15:40:04,782 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 15:40:04,782 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42433.77 MB 2025-02-14 15:40:04,782 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47152.37 MB 2025-02-14 15:40:04,782 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 15:40:04,782 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43856.09 MB 2025-02-14 15:40:04,783 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 15:40:04,783 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 15:40:04,783 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 15:40:04,783 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:40:04,783 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34180.42 MB 2025-02-14 15:40:04,783 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38311.81 MB 2025-02-14 15:40:04,783 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 15:40:04,783 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42433.77 MB 2025-02-14 15:40:04,783 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47152.37 MB 2025-02-14 15:40:04,783 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 15:40:04,783 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43856.09 MB 2025-02-14 15:40:04,943 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 15:40:04,943 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 15:40:04,943 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 15:40:04,943 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:40:04,943 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39019.60 MB 2025-02-14 15:40:04,943 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39786.60 MB 2025-02-14 15:40:04,943 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 15:40:04,943 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47152.37 MB 2025-02-14 15:40:04,943 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47565.50 MB 2025-02-14 15:40:04,943 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 15:40:04,943 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40494.39 MB 2025-02-14 15:40:04,960 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 15:40:04,960 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 15:40:04,960 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:40:04,960 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:40:04,960 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40199.49 MB 2025-02-14 15:40:04,960 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40404.66 MB 2025-02-14 15:40:04,960 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.17 MB 2025-02-14 15:40:04,960 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47565.50 MB 2025-02-14 15:40:04,960 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47565.50 MB 2025-02-14 15:40:04,960 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:40:04,960 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40632.19 MB 2025-02-14 15:40:04,961 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 15:40:04,961 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 15:40:04,961 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.87 seconds 2025-02-14 15:40:04,961 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:40:04,961 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28560.00 MB 2025-02-14 15:40:04,961 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40604.58 MB 2025-02-14 15:40:04,961 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12044.57 MB 2025-02-14 15:40:04,961 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43308.29 MB 2025-02-14 15:40:04,961 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47565.50 MB 2025-02-14 15:40:04,961 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4257.22 MB 2025-02-14 15:40:04,961 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40632.19 MB 2025-02-14 15:40:05,227 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 15:40:05,227 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 15:40:05,227 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 15:40:05,227 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:40:05,227 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40604.58 MB 2025-02-14 15:40:05,227 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40704.46 MB 2025-02-14 15:40:05,227 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 99.89 MB 2025-02-14 15:40:05,227 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47565.50 MB 2025-02-14 15:40:05,227 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47565.50 MB 2025-02-14 15:40:05,227 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:40:05,227 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41303.80 MB 2025-02-14 15:40:05,245 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8115, cut from 8117 2025-02-14 15:40:05,245 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 15:40:05,251 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 15:40:05,252 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 15:40:05,252 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:40:05,252 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:40:05,252 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29821.77 MB 2025-02-14 15:40:05,252 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33992.14 MB 2025-02-14 15:40:05,252 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4170.37 MB 2025-02-14 15:40:05,252 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47565.50 MB 2025-02-14 15:40:05,252 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51736.74 MB 2025-02-14 15:40:05,252 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4171.24 MB 2025-02-14 15:40:05,252 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38162.00 MB 2025-02-14 15:40:05,405 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7907] 2025-02-14 15:40:05,406 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:40:05,406 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:40:05,407 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:40:05,407 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 15:40:05,411 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 15:40:05,412 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:40:05,412 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 15:40:05,412 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 15:40:05,413 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:40:05,413 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:40:05,414 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:40:05,414 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:40:05,419 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 15:40:05,420 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:40:05,420 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:40:05,420 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:40:05,420 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:40:05,420 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 15:40:05,421 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:40:05,421 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:40:05,421 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:40:05,421 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 15:40:05,421 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 15:40:05,422 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:40:05,422 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:40:05,426 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:40:05,426 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:40:05,427 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:40:05,427 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:40:05,428 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:40:05,428 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:40:05,431 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:40:05,431 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:40:13,120 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:40:13,120 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:40:13,125 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 15:40:13,126 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:40:13,126 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 185, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 15:40:13,127 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:40:13,127 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 185, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 15:40:16,005 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 15:40:16,005 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 15:40:16,005 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.87 seconds 2025-02-14 15:40:16,005 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:40:16,005 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25337.38 MB 2025-02-14 15:40:16,005 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25992.08 MB 2025-02-14 15:40:16,005 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 654.70 MB 2025-02-14 15:40:16,005 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51736.74 MB 2025-02-14 15:40:16,005 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30907.83 MB 2025-02-14 15:40:16,005 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20828.91 MB 2025-02-14 15:40:16,005 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34809.55 MB 2025-02-14 15:40:16,019 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 15:40:16,019 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 15:40:16,019 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:40:16,019 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:40:16,019 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25992.08 MB 2025-02-14 15:40:16,019 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26267.15 MB 2025-02-14 15:40:16,019 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 275.06 MB 2025-02-14 15:40:16,019 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30907.83 MB 2025-02-14 15:40:16,019 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31543.26 MB 2025-02-14 15:40:16,019 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 635.44 MB 2025-02-14 15:40:16,019 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28538.24 MB 2025-02-14 15:40:16,889 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 15:40:16,889 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 15:40:16,889 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.87 seconds 2025-02-14 15:40:16,889 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:40:16,889 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26267.15 MB 2025-02-14 15:40:16,889 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26504.70 MB 2025-02-14 15:40:16,889 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 237.55 MB 2025-02-14 15:40:16,890 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31543.26 MB 2025-02-14 15:40:16,890 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30599.54 MB 2025-02-14 15:40:16,890 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -943.72 MB 2025-02-14 15:40:16,890 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30438.62 MB 2025-02-14 15:40:16,901 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 15:40:16,901 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 15:40:16,901 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:40:16,901 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:40:16,901 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26504.63 MB 2025-02-14 15:40:16,901 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27349.99 MB 2025-02-14 15:40:16,901 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 845.36 MB 2025-02-14 15:40:16,901 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30599.54 MB 2025-02-14 15:40:16,901 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30599.54 MB 2025-02-14 15:40:16,901 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:40:16,901 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27984.30 MB 2025-02-14 15:40:16,995 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 15:40:16,995 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 15:40:16,995 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 15:40:16,995 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:40:16,995 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27349.99 MB 2025-02-14 15:40:16,995 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28353.28 MB 2025-02-14 15:40:16,995 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1003.28 MB 2025-02-14 15:40:16,995 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30599.54 MB 2025-02-14 15:40:16,995 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32717.67 MB 2025-02-14 15:40:16,995 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2118.12 MB 2025-02-14 15:40:16,995 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30834.31 MB 2025-02-14 15:40:16,996 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 15:40:16,996 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 15:40:16,996 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 15:40:16,996 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:40:16,996 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26504.63 MB 2025-02-14 15:40:16,996 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28353.28 MB 2025-02-14 15:40:16,996 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1848.65 MB 2025-02-14 15:40:16,996 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30599.54 MB 2025-02-14 15:40:16,996 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32717.67 MB 2025-02-14 15:40:16,996 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2118.12 MB 2025-02-14 15:40:16,996 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30834.31 MB 2025-02-14 15:40:17,068 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 15:40:17,068 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 15:40:17,069 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 15:40:17,069 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:40:17,069 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28670.01 MB 2025-02-14 15:40:17,069 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29013.25 MB 2025-02-14 15:40:17,069 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 343.23 MB 2025-02-14 15:40:17,069 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32717.67 MB 2025-02-14 15:40:17,069 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32900.12 MB 2025-02-14 15:40:17,069 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 182.45 MB 2025-02-14 15:40:17,069 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29335.63 MB 2025-02-14 15:40:17,078 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 15:40:17,078 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 15:40:17,078 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:40:17,078 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:40:17,078 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29198.02 MB 2025-02-14 15:40:17,078 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29399.71 MB 2025-02-14 15:40:17,078 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 201.68 MB 2025-02-14 15:40:17,078 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32900.12 MB 2025-02-14 15:40:17,078 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32902.22 MB 2025-02-14 15:40:17,078 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-14 15:40:17,078 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29425.98 MB 2025-02-14 15:40:17,079 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 15:40:17,079 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 15:40:17,079 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.95 seconds 2025-02-14 15:40:17,079 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:40:17,079 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24692.82 MB 2025-02-14 15:40:17,079 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29601.18 MB 2025-02-14 15:40:17,079 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4908.36 MB 2025-02-14 15:40:17,079 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51736.74 MB 2025-02-14 15:40:17,079 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32902.22 MB 2025-02-14 15:40:17,079 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18834.52 MB 2025-02-14 15:40:17,079 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29601.18 MB 2025-02-14 15:40:17,342 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 15:40:17,342 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 15:40:17,342 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 15:40:17,342 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:40:17,342 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29601.18 MB 2025-02-14 15:40:17,342 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29701.50 MB 2025-02-14 15:40:17,342 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.32 MB 2025-02-14 15:40:17,342 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32902.22 MB 2025-02-14 15:40:17,342 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32902.22 MB 2025-02-14 15:40:17,342 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:40:17,342 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30303.42 MB 2025-02-14 15:40:17,360 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8150, cut from 8152 2025-02-14 15:40:17,360 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-14 15:40:17,366 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 15:40:17,366 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 15:40:17,366 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:40:17,366 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:40:17,366 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25369.44 MB 2025-02-14 15:40:17,366 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29557.77 MB 2025-02-14 15:40:17,366 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4188.33 MB 2025-02-14 15:40:17,366 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32902.22 MB 2025-02-14 15:40:17,366 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43375.39 MB 2025-02-14 15:40:17,366 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10473.18 MB 2025-02-14 15:40:17,366 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33745.78 MB 2025-02-14 15:40:17,517 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7942] 2025-02-14 15:40:17,519 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:40:17,519 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:40:17,519 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:40:17,519 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 15:40:17,524 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 15:40:17,525 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:40:17,525 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 15:40:17,525 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-14 15:40:17,526 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:40:17,526 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:40:17,526 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:40:17,526 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:40:17,532 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 15:40:17,532 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:40:17,532 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:40:17,533 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:40:17,533 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:40:17,533 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 15:40:17,533 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:40:17,533 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:40:17,534 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:40:17,534 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 15:40:17,534 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 15:40:17,534 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:40:17,534 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:40:17,537 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:40:17,537 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:40:17,538 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:40:17,538 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:40:17,539 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:40:17,539 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:40:17,542 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:40:17,542 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:40:25,278 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:40:25,278 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:40:25,283 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 15:40:25,284 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:40:25,284 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 151, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 15:40:25,285 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:40:25,285 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 151, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 15:40:27,636 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 15:40:27,636 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 15:40:27,636 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.35 seconds 2025-02-14 15:40:27,636 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:40:27,636 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25221.56 MB 2025-02-14 15:40:27,636 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25755.94 MB 2025-02-14 15:40:27,636 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 534.38 MB 2025-02-14 15:40:27,636 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43375.39 MB 2025-02-14 15:40:27,636 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34546.38 MB 2025-02-14 15:40:27,636 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8829.01 MB 2025-02-14 15:40:27,636 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34692.93 MB 2025-02-14 15:40:27,648 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 15:40:27,648 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 15:40:27,648 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:40:27,648 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:40:27,648 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25755.94 MB 2025-02-14 15:40:27,648 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25972.71 MB 2025-02-14 15:40:27,648 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 216.77 MB 2025-02-14 15:40:27,648 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34546.38 MB 2025-02-14 15:40:27,648 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34546.38 MB 2025-02-14 15:40:27,648 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:40:27,648 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27817.60 MB 2025-02-14 15:40:28,350 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 15:40:28,350 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 15:40:28,350 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.70 seconds 2025-02-14 15:40:28,350 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:40:28,350 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25972.71 MB 2025-02-14 15:40:28,351 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26165.14 MB 2025-02-14 15:40:28,351 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 192.43 MB 2025-02-14 15:40:28,351 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34546.38 MB 2025-02-14 15:40:28,351 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34122.76 MB 2025-02-14 15:40:28,351 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -423.62 MB 2025-02-14 15:40:28,351 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30143.14 MB 2025-02-14 15:40:28,357 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 15:40:28,357 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 15:40:28,357 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 15:40:28,357 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:40:28,357 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26165.07 MB 2025-02-14 15:40:28,357 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26849.86 MB 2025-02-14 15:40:28,357 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 684.79 MB 2025-02-14 15:40:28,357 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34122.76 MB 2025-02-14 15:40:28,357 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34122.76 MB 2025-02-14 15:40:28,357 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:40:28,357 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27363.68 MB 2025-02-14 15:40:28,434 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 15:40:28,434 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 15:40:28,434 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 15:40:28,434 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:40:28,434 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26849.86 MB 2025-02-14 15:40:28,434 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27662.58 MB 2025-02-14 15:40:28,434 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 812.71 MB 2025-02-14 15:40:28,434 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34122.76 MB 2025-02-14 15:40:28,434 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34122.76 MB 2025-02-14 15:40:28,434 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:40:28,434 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29672.34 MB 2025-02-14 15:40:28,434 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 15:40:28,434 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 15:40:28,434 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 15:40:28,434 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:40:28,434 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26165.07 MB 2025-02-14 15:40:28,434 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27662.58 MB 2025-02-14 15:40:28,434 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1497.50 MB 2025-02-14 15:40:28,434 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34122.76 MB 2025-02-14 15:40:28,435 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34122.76 MB 2025-02-14 15:40:28,435 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:40:28,435 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29672.34 MB 2025-02-14 15:40:28,491 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 15:40:28,491 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 15:40:28,491 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-14 15:40:28,491 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:40:28,491 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27919.15 MB 2025-02-14 15:40:28,491 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28197.19 MB 2025-02-14 15:40:28,491 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 278.04 MB 2025-02-14 15:40:28,491 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34122.76 MB 2025-02-14 15:40:28,492 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34269.56 MB 2025-02-14 15:40:28,492 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 146.80 MB 2025-02-14 15:40:28,492 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28465.68 MB 2025-02-14 15:40:28,499 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 15:40:28,499 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 15:40:28,499 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:40:28,499 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:40:28,499 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28347.17 MB 2025-02-14 15:40:28,499 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28553.53 MB 2025-02-14 15:40:28,499 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.36 MB 2025-02-14 15:40:28,499 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34269.56 MB 2025-02-14 15:40:28,499 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34269.56 MB 2025-02-14 15:40:28,499 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:40:28,499 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28562.58 MB 2025-02-14 15:40:28,500 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 15:40:28,500 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 15:40:28,500 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.21 seconds 2025-02-14 15:40:28,500 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:40:28,500 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24695.46 MB 2025-02-14 15:40:28,500 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28754.53 MB 2025-02-14 15:40:28,500 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4059.07 MB 2025-02-14 15:40:28,501 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43375.39 MB 2025-02-14 15:40:28,501 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34269.56 MB 2025-02-14 15:40:28,501 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9105.83 MB 2025-02-14 15:40:28,501 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28754.53 MB 2025-02-14 15:40:28,762 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 15:40:28,762 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 15:40:28,762 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 15:40:28,762 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:40:28,762 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28754.53 MB 2025-02-14 15:40:28,762 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28854.96 MB 2025-02-14 15:40:28,762 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.43 MB 2025-02-14 15:40:28,762 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34269.56 MB 2025-02-14 15:40:28,762 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34269.56 MB 2025-02-14 15:40:28,762 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:40:28,762 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29457.54 MB 2025-02-14 15:40:28,780 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8159, cut from 8161 2025-02-14 15:40:28,781 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-14 15:40:28,787 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 15:40:28,787 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 15:40:28,787 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:40:28,787 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:40:28,787 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28854.96 MB 2025-02-14 15:40:28,787 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33047.91 MB 2025-02-14 15:40:28,787 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4192.95 MB 2025-02-14 15:40:28,787 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34269.56 MB 2025-02-14 15:40:28,787 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42658.17 MB 2025-02-14 15:40:28,787 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8388.61 MB 2025-02-14 15:40:28,787 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37240.34 MB 2025-02-14 15:40:28,940 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7951] 2025-02-14 15:40:28,941 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:40:28,941 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:40:28,942 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:40:28,942 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 15:40:28,946 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 15:40:28,947 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:40:28,947 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 15:40:28,947 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-14 15:40:28,948 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:40:28,948 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:40:28,949 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:40:28,949 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:40:28,955 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 15:40:28,955 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:40:28,955 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:40:28,956 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:40:28,956 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:40:28,956 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 15:40:28,957 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:40:28,957 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:40:28,957 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:40:28,957 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 15:40:28,957 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 15:40:28,958 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:40:28,958 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:40:28,962 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:40:28,962 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:40:28,962 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:40:28,963 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:40:28,963 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:40:28,963 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:40:28,966 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:40:28,966 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:41:18,749 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:41:18,749 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:41:18,754 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 15:41:18,755 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:41:18,755 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 158, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 15:41:18,756 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:41:18,756 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 158, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 15:41:21,225 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 15:41:21,225 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 15:41:21,225 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.46 seconds 2025-02-14 15:41:21,225 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:41:21,225 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25392.43 MB 2025-02-14 15:41:21,225 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25951.58 MB 2025-02-14 15:41:21,225 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 559.15 MB 2025-02-14 15:41:21,225 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42658.17 MB 2025-02-14 15:41:21,225 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29934.75 MB 2025-02-14 15:41:21,225 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12723.42 MB 2025-02-14 15:41:21,225 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34865.13 MB 2025-02-14 15:41:21,242 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 15:41:21,242 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 15:41:21,242 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:41:21,242 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:41:21,242 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25951.58 MB 2025-02-14 15:41:21,242 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26222.49 MB 2025-02-14 15:41:21,242 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 270.91 MB 2025-02-14 15:41:21,242 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29934.75 MB 2025-02-14 15:41:21,242 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29934.75 MB 2025-02-14 15:41:21,242 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:41:21,242 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28220.46 MB 2025-02-14 15:41:22,041 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 15:41:22,041 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 15:41:22,041 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.80 seconds 2025-02-14 15:41:22,041 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:41:22,041 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26222.49 MB 2025-02-14 15:41:22,041 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26432.17 MB 2025-02-14 15:41:22,041 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 209.68 MB 2025-02-14 15:41:22,041 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29934.75 MB 2025-02-14 15:41:22,041 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29511.12 MB 2025-02-14 15:41:22,041 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -423.62 MB 2025-02-14 15:41:22,041 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30393.96 MB 2025-02-14 15:41:22,052 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 15:41:22,052 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 15:41:22,052 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:41:22,052 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:41:22,052 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26432.10 MB 2025-02-14 15:41:22,052 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27178.29 MB 2025-02-14 15:41:22,053 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 746.18 MB 2025-02-14 15:41:22,053 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29511.12 MB 2025-02-14 15:41:22,053 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29511.12 MB 2025-02-14 15:41:22,053 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:41:22,053 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27738.18 MB 2025-02-14 15:41:22,165 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 15:41:22,165 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 15:41:22,165 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 15:41:22,165 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:41:22,165 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27178.29 MB 2025-02-14 15:41:22,165 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28064.39 MB 2025-02-14 15:41:22,165 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 886.10 MB 2025-02-14 15:41:22,165 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29511.12 MB 2025-02-14 15:41:22,165 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31750.88 MB 2025-02-14 15:41:22,165 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2239.76 MB 2025-02-14 15:41:22,165 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30256.43 MB 2025-02-14 15:41:22,166 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 15:41:22,166 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 15:41:22,166 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 15:41:22,166 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:41:22,166 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26432.10 MB 2025-02-14 15:41:22,166 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28064.39 MB 2025-02-14 15:41:22,167 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1632.28 MB 2025-02-14 15:41:22,167 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29511.12 MB 2025-02-14 15:41:22,167 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31750.88 MB 2025-02-14 15:41:22,167 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2239.76 MB 2025-02-14 15:41:22,167 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30256.43 MB 2025-02-14 15:41:22,268 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 15:41:22,269 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 15:41:22,269 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 15:41:22,269 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:41:22,269 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28343.96 MB 2025-02-14 15:41:22,269 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28646.93 MB 2025-02-14 15:41:22,269 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 302.97 MB 2025-02-14 15:41:22,269 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31750.88 MB 2025-02-14 15:41:22,269 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31912.36 MB 2025-02-14 15:41:22,269 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 161.48 MB 2025-02-14 15:41:22,269 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28935.17 MB 2025-02-14 15:41:22,286 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 15:41:22,286 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 15:41:22,286 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:41:22,286 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:41:22,286 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28810.03 MB 2025-02-14 15:41:22,286 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29011.31 MB 2025-02-14 15:41:22,286 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 201.28 MB 2025-02-14 15:41:22,286 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31912.36 MB 2025-02-14 15:41:22,286 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31912.36 MB 2025-02-14 15:41:22,286 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:41:22,286 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29023.70 MB 2025-02-14 15:41:22,288 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 15:41:22,289 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 15:41:22,289 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.53 seconds 2025-02-14 15:41:22,289 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:41:22,289 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24841.94 MB 2025-02-14 15:41:22,289 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29212.30 MB 2025-02-14 15:41:22,289 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4370.36 MB 2025-02-14 15:41:22,289 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42658.17 MB 2025-02-14 15:41:22,289 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31912.36 MB 2025-02-14 15:41:22,289 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10745.81 MB 2025-02-14 15:41:22,289 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29212.30 MB 2025-02-14 15:41:22,568 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 15:41:22,568 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 15:41:22,568 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 15:41:22,568 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:41:22,568 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29212.30 MB 2025-02-14 15:41:22,568 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29312.73 MB 2025-02-14 15:41:22,568 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.43 MB 2025-02-14 15:41:22,568 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31912.36 MB 2025-02-14 15:41:22,568 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31912.36 MB 2025-02-14 15:41:22,568 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:41:22,568 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29915.31 MB 2025-02-14 15:41:22,588 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8159, cut from 8161 2025-02-14 15:41:22,588 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-14 15:41:22,595 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 15:41:22,596 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 15:41:22,596 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:41:22,596 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:41:22,596 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25462.33 MB 2025-02-14 15:41:22,596 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29655.28 MB 2025-02-14 15:41:22,596 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4192.95 MB 2025-02-14 15:41:22,596 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31912.36 MB 2025-02-14 15:41:22,596 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42398.12 MB 2025-02-14 15:41:22,596 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10485.76 MB 2025-02-14 15:41:22,596 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33847.71 MB 2025-02-14 15:41:22,831 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7951] 2025-02-14 15:41:22,833 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:41:22,833 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:41:22,835 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:41:22,835 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 15:41:22,842 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 15:41:22,844 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:41:22,844 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 15:41:22,844 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-14 15:41:22,845 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:41:22,845 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:41:22,846 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:41:22,847 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:41:22,855 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 15:41:22,856 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:41:22,856 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:41:22,857 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:41:22,857 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:41:22,857 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 15:41:22,858 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:41:22,858 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:41:22,859 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:41:22,859 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 15:41:22,859 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 15:41:22,860 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:41:22,860 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:41:22,871 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:41:22,871 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:41:22,874 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:41:22,874 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:41:22,877 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:41:22,877 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:41:22,882 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:41:22,882 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:42:03,231 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:42:03,232 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:42:03,237 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 15:42:03,238 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:42:03,239 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1238, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 15:42:03,240 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:42:03,240 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1238, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 15:42:22,301 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 15:42:22,301 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 15:42:22,301 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.05 seconds 2025-02-14 15:42:22,301 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:42:22,301 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33039.45 MB 2025-02-14 15:42:22,301 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37420.67 MB 2025-02-14 15:42:22,301 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4381.21 MB 2025-02-14 15:42:22,301 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46646.95 MB 2025-02-14 15:42:22,301 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46531.61 MB 2025-02-14 15:42:22,301 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -115.34 MB 2025-02-14 15:42:22,301 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46361.19 MB 2025-02-14 15:42:22,375 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 15:42:22,375 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 15:42:22,375 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 15:42:22,375 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:42:22,375 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37420.67 MB 2025-02-14 15:42:22,375 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33657.99 MB 2025-02-14 15:42:22,375 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3762.67 MB 2025-02-14 15:42:22,375 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46531.61 MB 2025-02-14 15:42:22,375 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55180.26 MB 2025-02-14 15:42:22,375 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8648.65 MB 2025-02-14 15:42:22,375 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50444.44 MB 2025-02-14 15:42:24,281 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 15:42:24,281 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 15:42:24,281 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.90 seconds 2025-02-14 15:42:24,281 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:42:24,281 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33657.99 MB 2025-02-14 15:42:24,281 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34188.83 MB 2025-02-14 15:42:24,281 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 15:42:24,281 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55180.26 MB 2025-02-14 15:42:24,281 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42148.56 MB 2025-02-14 15:42:24,281 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13031.70 MB 2025-02-14 15:42:24,281 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38168.17 MB 2025-02-14 15:42:24,295 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 15:42:24,295 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 15:42:24,295 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:42:24,295 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:42:24,295 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34188.83 MB 2025-02-14 15:42:24,295 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36078.00 MB 2025-02-14 15:42:24,295 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.17 MB 2025-02-14 15:42:24,295 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42148.56 MB 2025-02-14 15:42:24,295 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42148.56 MB 2025-02-14 15:42:24,295 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:42:24,295 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37495.43 MB 2025-02-14 15:42:24,505 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 15:42:24,505 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 15:42:24,505 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 15:42:24,505 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:42:24,505 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36078.00 MB 2025-02-14 15:42:24,505 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38319.86 MB 2025-02-14 15:42:24,505 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 15:42:24,505 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42148.56 MB 2025-02-14 15:42:24,506 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46867.15 MB 2025-02-14 15:42:24,506 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 15:42:24,506 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43864.14 MB 2025-02-14 15:42:24,506 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 15:42:24,506 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 15:42:24,506 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 15:42:24,506 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:42:24,506 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34188.83 MB 2025-02-14 15:42:24,506 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38319.86 MB 2025-02-14 15:42:24,506 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.02 MB 2025-02-14 15:42:24,506 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42148.56 MB 2025-02-14 15:42:24,506 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46867.15 MB 2025-02-14 15:42:24,506 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 15:42:24,506 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43864.14 MB 2025-02-14 15:42:24,674 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 15:42:24,675 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 15:42:24,675 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 15:42:24,675 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:42:24,675 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39027.64 MB 2025-02-14 15:42:24,675 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39794.65 MB 2025-02-14 15:42:24,675 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 15:42:24,675 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46867.15 MB 2025-02-14 15:42:24,675 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47282.39 MB 2025-02-14 15:42:24,675 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 15:42:24,675 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40502.43 MB 2025-02-14 15:42:24,693 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 15:42:24,693 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 15:42:24,693 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:42:24,693 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:42:24,693 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40207.53 MB 2025-02-14 15:42:24,693 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40412.42 MB 2025-02-14 15:42:24,693 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 204.89 MB 2025-02-14 15:42:24,693 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47282.39 MB 2025-02-14 15:42:24,693 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47282.39 MB 2025-02-14 15:42:24,693 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:42:24,693 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40631.06 MB 2025-02-14 15:42:24,694 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 15:42:24,694 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 15:42:24,694 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.45 seconds 2025-02-14 15:42:24,694 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:42:24,694 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28726.16 MB 2025-02-14 15:42:24,694 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40612.54 MB 2025-02-14 15:42:24,694 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11886.38 MB 2025-02-14 15:42:24,694 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44522.54 MB 2025-02-14 15:42:24,694 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47282.39 MB 2025-02-14 15:42:24,694 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2759.85 MB 2025-02-14 15:42:24,694 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40631.06 MB 2025-02-14 15:42:24,959 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 15:42:24,959 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 15:42:24,959 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 15:42:24,959 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:42:24,959 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40612.54 MB 2025-02-14 15:42:24,959 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40712.53 MB 2025-02-14 15:42:24,959 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 99.99 MB 2025-02-14 15:42:24,959 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47282.39 MB 2025-02-14 15:42:24,959 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47282.39 MB 2025-02-14 15:42:24,959 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:42:24,959 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41312.45 MB 2025-02-14 15:42:24,977 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8123, cut from 8125 2025-02-14 15:42:24,977 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 15:42:24,983 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 15:42:24,983 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 15:42:24,983 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:42:24,983 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:42:24,983 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29988.12 MB 2025-02-14 15:42:24,983 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34163.55 MB 2025-02-14 15:42:24,983 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4175.43 MB 2025-02-14 15:42:24,983 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47282.39 MB 2025-02-14 15:42:24,983 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55633.25 MB 2025-02-14 15:42:24,983 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8350.86 MB 2025-02-14 15:42:24,983 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38337.52 MB 2025-02-14 15:42:25,145 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7915] 2025-02-14 15:42:25,147 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:42:25,147 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:42:25,147 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:42:25,148 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 15:42:25,152 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 15:42:25,153 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:42:25,153 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 15:42:25,153 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 15:42:25,154 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:42:25,154 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:42:25,155 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:42:25,155 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:42:25,161 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 15:42:25,161 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:42:25,161 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:42:25,162 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:42:25,162 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:42:25,162 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 15:42:25,162 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:42:25,162 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:42:25,163 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:42:25,163 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 15:42:25,163 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 15:42:25,163 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:42:25,163 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:42:25,168 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:42:25,169 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:42:25,170 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:42:25,170 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:42:25,171 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:42:25,171 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:42:25,175 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:42:25,175 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:43:24,571 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:43:24,571 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:43:24,579 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 15:43:24,581 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:43:24,581 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1042, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 15:43:24,583 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:43:24,583 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1042, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 15:43:40,655 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 15:43:40,656 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 15:43:40,656 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.06 seconds 2025-02-14 15:43:40,656 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:43:40,656 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31795.29 MB 2025-02-14 15:43:40,656 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35482.87 MB 2025-02-14 15:43:40,656 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3687.58 MB 2025-02-14 15:43:40,656 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55633.25 MB 2025-02-14 15:43:40,656 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41628.47 MB 2025-02-14 15:43:40,656 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14004.78 MB 2025-02-14 15:43:40,656 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44438.36 MB 2025-02-14 15:43:40,726 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 15:43:40,726 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 15:43:40,727 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 15:43:40,727 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:43:40,727 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35482.87 MB 2025-02-14 15:43:40,727 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32760.64 MB 2025-02-14 15:43:40,727 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2722.23 MB 2025-02-14 15:43:40,727 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41628.47 MB 2025-02-14 15:43:40,727 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53389.30 MB 2025-02-14 15:43:40,727 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 11760.83 MB 2025-02-14 15:43:40,727 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47026.36 MB 2025-02-14 15:43:42,646 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 15:43:42,646 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 15:43:42,646 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 15:43:42,646 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:43:42,646 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32760.64 MB 2025-02-14 15:43:42,646 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33291.48 MB 2025-02-14 15:43:42,646 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 15:43:42,646 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53389.30 MB 2025-02-14 15:43:42,646 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37230.74 MB 2025-02-14 15:43:42,646 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16158.56 MB 2025-02-14 15:43:42,646 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37271.86 MB 2025-02-14 15:43:42,660 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 15:43:42,660 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 15:43:42,660 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:43:42,660 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:43:42,660 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33291.48 MB 2025-02-14 15:43:42,660 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35181.02 MB 2025-02-14 15:43:42,660 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 15:43:42,660 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37230.74 MB 2025-02-14 15:43:42,660 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40061.89 MB 2025-02-14 15:43:42,660 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-14 15:43:42,660 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36598.45 MB 2025-02-14 15:43:42,871 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 15:43:42,871 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 15:43:42,871 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 15:43:42,871 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:43:42,871 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35181.02 MB 2025-02-14 15:43:42,871 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37422.87 MB 2025-02-14 15:43:42,871 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 15:43:42,871 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40061.89 MB 2025-02-14 15:43:42,871 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46196.06 MB 2025-02-14 15:43:42,871 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 15:43:42,871 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42967.16 MB 2025-02-14 15:43:42,872 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 15:43:42,872 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 15:43:42,872 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 15:43:42,872 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:43:42,872 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33291.48 MB 2025-02-14 15:43:42,872 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37422.87 MB 2025-02-14 15:43:42,872 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 15:43:42,872 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37230.74 MB 2025-02-14 15:43:42,872 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46196.06 MB 2025-02-14 15:43:42,872 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8965.32 MB 2025-02-14 15:43:42,872 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42967.16 MB 2025-02-14 15:43:43,038 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 15:43:43,038 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 15:43:43,038 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 15:43:43,038 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:43:43,038 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38130.66 MB 2025-02-14 15:43:43,038 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38897.67 MB 2025-02-14 15:43:43,038 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 15:43:43,038 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46196.06 MB 2025-02-14 15:43:43,038 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46611.30 MB 2025-02-14 15:43:43,038 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 15:43:43,038 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39605.45 MB 2025-02-14 15:43:43,055 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 15:43:43,055 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 15:43:43,055 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:43:43,055 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:43:43,055 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39310.55 MB 2025-02-14 15:43:43,055 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39516.77 MB 2025-02-14 15:43:43,055 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.22 MB 2025-02-14 15:43:43,055 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46611.30 MB 2025-02-14 15:43:43,055 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46611.30 MB 2025-02-14 15:43:43,055 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:43:43,056 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39741.09 MB 2025-02-14 15:43:43,057 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 15:43:43,057 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 15:43:43,057 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.47 seconds 2025-02-14 15:43:43,057 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:43:43,057 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28164.88 MB 2025-02-14 15:43:43,057 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39717.82 MB 2025-02-14 15:43:43,057 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11552.94 MB 2025-02-14 15:43:43,057 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55633.25 MB 2025-02-14 15:43:43,057 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46611.30 MB 2025-02-14 15:43:43,057 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9021.95 MB 2025-02-14 15:43:43,057 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39741.09 MB 2025-02-14 15:43:43,325 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 15:43:43,325 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 15:43:43,325 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 15:43:43,325 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:43:43,325 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39717.82 MB 2025-02-14 15:43:43,325 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39818.27 MB 2025-02-14 15:43:43,325 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.45 MB 2025-02-14 15:43:43,325 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46611.30 MB 2025-02-14 15:43:43,325 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46611.30 MB 2025-02-14 15:43:43,325 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:43:43,325 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40421.93 MB 2025-02-14 15:43:43,343 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8161, cut from 8163 2025-02-14 15:43:43,343 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 15:43:43,350 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 15:43:43,350 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 15:43:43,350 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:43:43,350 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:43:43,350 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29427.77 MB 2025-02-14 15:43:43,350 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33622.08 MB 2025-02-14 15:43:43,350 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4194.31 MB 2025-02-14 15:43:43,350 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46611.30 MB 2025-02-14 15:43:43,350 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54999.91 MB 2025-02-14 15:43:43,350 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8388.61 MB 2025-02-14 15:43:43,350 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37816.38 MB 2025-02-14 15:43:43,512 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7953] 2025-02-14 15:43:43,513 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:43:43,513 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:43:43,514 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:43:43,514 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 15:43:43,519 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 15:43:43,520 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:43:43,520 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 15:43:43,520 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 15:43:43,521 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:43:43,521 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:43:43,521 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:43:43,521 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:43:43,527 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 15:43:43,528 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:43:43,528 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:43:43,528 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:43:43,528 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:43:43,528 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 15:43:43,529 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:43:43,529 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:43:43,529 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:43:43,529 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 15:43:43,530 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 15:43:43,530 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:43:43,530 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:43:43,536 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:43:43,536 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:43:43,538 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:43:43,538 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:43:43,539 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:43:43,539 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:43:43,543 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:43:43,543 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:43:51,165 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:43:51,165 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:43:51,170 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 15:43:51,171 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:43:51,171 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1267, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 15:43:51,172 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:43:51,172 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1267, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 15:44:10,769 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 15:44:10,770 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 15:44:10,770 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.59 seconds 2025-02-14 15:44:10,770 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:44:10,770 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33485.38 MB 2025-02-14 15:44:10,770 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37969.22 MB 2025-02-14 15:44:10,770 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4483.84 MB 2025-02-14 15:44:10,770 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59345.21 MB 2025-02-14 15:44:10,770 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50910.46 MB 2025-02-14 15:44:10,770 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8434.75 MB 2025-02-14 15:44:10,770 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46807.12 MB 2025-02-14 15:44:10,847 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 15:44:10,847 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 15:44:10,847 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 15:44:10,847 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:44:10,847 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37969.22 MB 2025-02-14 15:44:10,847 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34052.60 MB 2025-02-14 15:44:10,847 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3916.62 MB 2025-02-14 15:44:10,847 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50910.46 MB 2025-02-14 15:44:10,847 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 59798.19 MB 2025-02-14 15:44:10,847 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8887.73 MB 2025-02-14 15:44:10,847 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51341.78 MB 2025-02-14 15:44:12,772 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 15:44:12,772 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 15:44:12,772 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 15:44:12,772 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:44:12,772 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34052.60 MB 2025-02-14 15:44:12,772 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34583.45 MB 2025-02-14 15:44:12,772 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 15:44:12,773 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59798.19 MB 2025-02-14 15:44:12,773 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46424.65 MB 2025-02-14 15:44:12,773 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13373.54 MB 2025-02-14 15:44:12,773 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38562.78 MB 2025-02-14 15:44:12,786 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 15:44:12,786 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 15:44:12,786 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:44:12,786 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:44:12,786 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34583.45 MB 2025-02-14 15:44:12,786 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36472.80 MB 2025-02-14 15:44:12,786 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.35 MB 2025-02-14 15:44:12,786 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46424.65 MB 2025-02-14 15:44:12,786 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46424.65 MB 2025-02-14 15:44:12,786 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:44:12,786 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37890.23 MB 2025-02-14 15:44:13,002 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 15:44:13,002 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 15:44:13,002 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 15:44:13,002 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:44:13,002 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36472.80 MB 2025-02-14 15:44:13,002 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38714.66 MB 2025-02-14 15:44:13,002 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 15:44:13,002 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46424.65 MB 2025-02-14 15:44:13,002 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47840.23 MB 2025-02-14 15:44:13,002 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1415.58 MB 2025-02-14 15:44:13,002 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44258.94 MB 2025-02-14 15:44:13,002 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 15:44:13,002 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 15:44:13,003 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 15:44:13,003 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:44:13,003 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34583.45 MB 2025-02-14 15:44:13,003 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38714.66 MB 2025-02-14 15:44:13,003 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.21 MB 2025-02-14 15:44:13,003 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46424.65 MB 2025-02-14 15:44:13,003 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47840.23 MB 2025-02-14 15:44:13,003 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1415.58 MB 2025-02-14 15:44:13,003 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44258.94 MB 2025-02-14 15:44:13,169 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 15:44:13,169 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 15:44:13,169 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 15:44:13,169 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:44:13,169 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39422.44 MB 2025-02-14 15:44:13,169 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40189.45 MB 2025-02-14 15:44:13,169 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 15:44:13,169 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47840.23 MB 2025-02-14 15:44:13,169 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48255.47 MB 2025-02-14 15:44:13,169 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 15:44:13,169 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40897.23 MB 2025-02-14 15:44:13,187 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 15:44:13,187 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 15:44:13,187 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:44:13,187 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:44:13,187 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40602.34 MB 2025-02-14 15:44:13,187 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40809.14 MB 2025-02-14 15:44:13,187 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.81 MB 2025-02-14 15:44:13,187 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48255.47 MB 2025-02-14 15:44:13,187 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48255.47 MB 2025-02-14 15:44:13,187 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:44:13,187 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41036.76 MB 2025-02-14 15:44:13,188 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 15:44:13,188 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 15:44:13,188 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.01 seconds 2025-02-14 15:44:13,188 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:44:13,188 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29070.81 MB 2025-02-14 15:44:13,188 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41009.97 MB 2025-02-14 15:44:13,188 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11939.16 MB 2025-02-14 15:44:13,188 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57172.56 MB 2025-02-14 15:44:13,188 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48255.47 MB 2025-02-14 15:44:13,188 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8917.09 MB 2025-02-14 15:44:13,188 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41036.76 MB 2025-02-14 15:44:13,454 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 15:44:13,454 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 15:44:13,454 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 15:44:13,454 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:44:13,454 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41009.97 MB 2025-02-14 15:44:13,454 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41110.31 MB 2025-02-14 15:44:13,454 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.34 MB 2025-02-14 15:44:13,454 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48255.47 MB 2025-02-14 15:44:13,454 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48255.47 MB 2025-02-14 15:44:13,454 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:44:13,454 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41712.37 MB 2025-02-14 15:44:13,472 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8152, cut from 8154 2025-02-14 15:44:13,472 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 15:44:13,478 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 15:44:13,478 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 15:44:13,478 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:44:13,478 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:44:13,478 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30333.49 MB 2025-02-14 15:44:13,478 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34522.84 MB 2025-02-14 15:44:13,478 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4189.36 MB 2025-02-14 15:44:13,478 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48255.47 MB 2025-02-14 15:44:13,478 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52445.58 MB 2025-02-14 15:44:13,478 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4190.11 MB 2025-02-14 15:44:13,478 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38711.68 MB 2025-02-14 15:44:13,644 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7944] 2025-02-14 15:44:13,646 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:44:13,646 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:44:13,646 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:44:13,646 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 15:44:13,651 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 15:44:13,652 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:44:13,652 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 15:44:13,652 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 15:44:13,653 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:44:13,653 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:44:13,654 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:44:13,654 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:44:13,659 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 15:44:13,660 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:44:13,660 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:44:13,661 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:44:13,661 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:44:13,661 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 15:44:13,661 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:44:13,661 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:44:13,662 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:44:13,662 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 15:44:13,662 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 15:44:13,662 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:44:13,662 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:44:13,667 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:44:13,667 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:44:13,668 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:44:13,668 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:44:13,670 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:44:13,670 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:44:13,673 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:44:13,673 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:45:10,113 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:45:10,113 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:45:10,118 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 15:45:10,119 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:45:10,119 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 148, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 15:45:10,120 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:45:10,120 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 148, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 15:45:12,413 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 15:45:12,413 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 15:45:12,413 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.29 seconds 2025-02-14 15:45:12,413 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:45:12,413 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25809.12 MB 2025-02-14 15:45:12,413 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26332.89 MB 2025-02-14 15:45:12,413 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 523.76 MB 2025-02-14 15:45:12,413 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52445.58 MB 2025-02-14 15:45:12,413 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34177.29 MB 2025-02-14 15:45:12,413 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18268.29 MB 2025-02-14 15:45:12,413 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35280.49 MB 2025-02-14 15:45:12,424 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 15:45:12,425 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 15:45:12,425 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:45:12,425 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:45:12,425 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26332.89 MB 2025-02-14 15:45:12,425 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26565.58 MB 2025-02-14 15:45:12,425 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 232.69 MB 2025-02-14 15:45:12,425 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34177.29 MB 2025-02-14 15:45:12,425 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34177.29 MB 2025-02-14 15:45:12,425 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:45:12,425 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28384.44 MB 2025-02-14 15:45:13,137 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 15:45:13,137 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 15:45:13,137 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.71 seconds 2025-02-14 15:45:13,137 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:45:13,137 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26565.58 MB 2025-02-14 15:45:13,137 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26758.01 MB 2025-02-14 15:45:13,137 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 192.43 MB 2025-02-14 15:45:13,137 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34177.29 MB 2025-02-14 15:45:13,137 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34177.29 MB 2025-02-14 15:45:13,137 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:45:13,137 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30736.02 MB 2025-02-14 15:45:13,149 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 15:45:13,149 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 15:45:13,149 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:45:13,149 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:45:13,149 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26757.94 MB 2025-02-14 15:45:13,149 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27442.73 MB 2025-02-14 15:45:13,149 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 684.79 MB 2025-02-14 15:45:13,149 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34177.29 MB 2025-02-14 15:45:13,149 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34177.29 MB 2025-02-14 15:45:13,149 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:45:13,149 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27956.56 MB 2025-02-14 15:45:13,258 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 15:45:13,258 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 15:45:13,258 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 15:45:13,258 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:45:13,258 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27442.73 MB 2025-02-14 15:45:13,258 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28255.45 MB 2025-02-14 15:45:13,258 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 812.71 MB 2025-02-14 15:45:13,258 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34177.29 MB 2025-02-14 15:45:13,258 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34177.29 MB 2025-02-14 15:45:13,258 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:45:13,258 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30265.21 MB 2025-02-14 15:45:13,259 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 15:45:13,259 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 15:45:13,259 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 15:45:13,259 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:45:13,260 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26757.94 MB 2025-02-14 15:45:13,260 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28255.45 MB 2025-02-14 15:45:13,260 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1497.50 MB 2025-02-14 15:45:13,260 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34177.29 MB 2025-02-14 15:45:13,260 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34177.29 MB 2025-02-14 15:45:13,260 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:45:13,260 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30265.21 MB 2025-02-14 15:45:13,363 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 15:45:13,363 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 15:45:13,363 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 15:45:13,363 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:45:13,363 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28512.02 MB 2025-02-14 15:45:13,363 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28790.06 MB 2025-02-14 15:45:13,363 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 278.04 MB 2025-02-14 15:45:13,363 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34177.29 MB 2025-02-14 15:45:13,363 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34324.09 MB 2025-02-14 15:45:13,363 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 146.80 MB 2025-02-14 15:45:13,363 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29058.00 MB 2025-02-14 15:45:13,377 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 15:45:13,377 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 15:45:13,377 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:45:13,377 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:45:13,377 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28939.74 MB 2025-02-14 15:45:13,377 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29144.31 MB 2025-02-14 15:45:13,377 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 204.57 MB 2025-02-14 15:45:13,377 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34324.09 MB 2025-02-14 15:45:13,377 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34324.09 MB 2025-02-14 15:45:13,377 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:45:13,377 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29144.31 MB 2025-02-14 15:45:13,379 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 15:45:13,379 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 15:45:13,379 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.26 seconds 2025-02-14 15:45:13,379 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:45:13,379 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25293.48 MB 2025-02-14 15:45:13,379 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29344.94 MB 2025-02-14 15:45:13,379 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4051.47 MB 2025-02-14 15:45:13,379 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52445.58 MB 2025-02-14 15:45:13,379 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34324.09 MB 2025-02-14 15:45:13,379 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18121.49 MB 2025-02-14 15:45:13,379 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29344.94 MB 2025-02-14 15:45:13,658 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 15:45:13,658 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 15:45:13,658 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.28 seconds 2025-02-14 15:45:13,658 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:45:13,658 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29344.94 MB 2025-02-14 15:45:13,658 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29445.19 MB 2025-02-14 15:45:13,658 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.25 MB 2025-02-14 15:45:13,658 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34324.09 MB 2025-02-14 15:45:13,658 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34324.09 MB 2025-02-14 15:45:13,658 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:45:13,658 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30046.66 MB 2025-02-14 15:45:13,679 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8144, cut from 8146 2025-02-14 15:45:13,679 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-14 15:45:13,687 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 15:45:13,687 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 15:45:13,687 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:45:13,687 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:45:13,687 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25878.99 MB 2025-02-14 15:45:13,687 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30064.90 MB 2025-02-14 15:45:13,687 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4185.92 MB 2025-02-14 15:45:13,687 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34324.09 MB 2025-02-14 15:45:13,687 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42695.92 MB 2025-02-14 15:45:13,687 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8371.83 MB 2025-02-14 15:45:13,687 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34249.64 MB 2025-02-14 15:45:13,947 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7936] 2025-02-14 15:45:13,949 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:45:13,949 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:45:13,951 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:45:13,951 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 15:45:13,959 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 15:45:13,961 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:45:13,961 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 15:45:13,961 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-14 15:45:13,963 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:45:13,963 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:45:13,964 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:45:13,964 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:45:13,973 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 15:45:13,974 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:45:13,974 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:45:13,975 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:45:13,975 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:45:13,975 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 15:45:13,976 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:45:13,976 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:45:13,977 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:45:13,977 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 15:45:13,978 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 15:45:13,979 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:45:13,979 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:45:13,990 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:45:13,990 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:45:13,994 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:45:13,994 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:45:13,997 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:45:13,997 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:45:14,002 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:45:14,002 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:46:09,543 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:46:09,543 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:46:09,548 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 15:46:09,549 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:46:09,549 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1216, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 15:46:09,550 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:46:09,550 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1216, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 15:46:28,171 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 15:46:28,171 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 15:46:28,171 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.61 seconds 2025-02-14 15:46:28,171 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:46:28,171 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33372.71 MB 2025-02-14 15:46:28,171 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37676.07 MB 2025-02-14 15:46:28,171 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4303.36 MB 2025-02-14 15:46:28,171 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42695.92 MB 2025-02-14 15:46:28,171 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46380.61 MB 2025-02-14 15:46:28,171 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3684.70 MB 2025-02-14 15:46:28,171 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46694.45 MB 2025-02-14 15:46:28,242 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 15:46:28,243 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 15:46:28,243 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 15:46:28,243 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:46:28,243 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37676.07 MB 2025-02-14 15:46:28,243 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34030.18 MB 2025-02-14 15:46:28,243 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3645.89 MB 2025-02-14 15:46:28,243 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46380.61 MB 2025-02-14 15:46:28,243 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54926.51 MB 2025-02-14 15:46:28,243 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8545.89 MB 2025-02-14 15:46:28,243 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50408.69 MB 2025-02-14 15:46:30,184 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 15:46:30,184 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 15:46:30,185 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-14 15:46:30,185 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:46:30,185 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34030.18 MB 2025-02-14 15:46:30,185 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34561.02 MB 2025-02-14 15:46:30,185 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 15:46:30,185 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54926.51 MB 2025-02-14 15:46:30,185 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37891.34 MB 2025-02-14 15:46:30,185 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17035.17 MB 2025-02-14 15:46:30,185 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38540.35 MB 2025-02-14 15:46:30,199 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 15:46:30,199 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 15:46:30,199 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:46:30,199 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:46:30,199 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34561.02 MB 2025-02-14 15:46:30,199 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36450.37 MB 2025-02-14 15:46:30,199 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.35 MB 2025-02-14 15:46:30,199 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37891.34 MB 2025-02-14 15:46:30,199 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40722.50 MB 2025-02-14 15:46:30,199 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-14 15:46:30,199 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37867.80 MB 2025-02-14 15:46:30,407 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 15:46:30,407 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 15:46:30,407 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 15:46:30,407 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:46:30,407 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36450.37 MB 2025-02-14 15:46:30,407 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38692.23 MB 2025-02-14 15:46:30,407 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 15:46:30,407 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40722.50 MB 2025-02-14 15:46:30,407 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47328.53 MB 2025-02-14 15:46:30,407 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 15:46:30,407 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44236.51 MB 2025-02-14 15:46:30,408 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 15:46:30,408 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 15:46:30,408 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 15:46:30,408 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:46:30,408 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34561.02 MB 2025-02-14 15:46:30,408 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38692.23 MB 2025-02-14 15:46:30,408 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.21 MB 2025-02-14 15:46:30,408 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37891.34 MB 2025-02-14 15:46:30,408 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47328.53 MB 2025-02-14 15:46:30,408 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9437.18 MB 2025-02-14 15:46:30,408 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44236.51 MB 2025-02-14 15:46:30,566 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 15:46:30,566 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 15:46:30,566 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 15:46:30,566 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:46:30,566 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39400.02 MB 2025-02-14 15:46:30,566 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40167.02 MB 2025-02-14 15:46:30,566 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 15:46:30,566 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47328.53 MB 2025-02-14 15:46:30,566 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47743.76 MB 2025-02-14 15:46:30,566 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 15:46:30,566 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40874.81 MB 2025-02-14 15:46:30,583 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 15:46:30,583 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 15:46:30,583 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:46:30,583 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:46:30,583 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40579.91 MB 2025-02-14 15:46:30,583 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40785.06 MB 2025-02-14 15:46:30,583 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.15 MB 2025-02-14 15:46:30,583 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47743.76 MB 2025-02-14 15:46:30,583 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47743.76 MB 2025-02-14 15:46:30,583 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:46:30,583 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41009.85 MB 2025-02-14 15:46:30,584 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 15:46:30,584 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 15:46:30,584 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.03 seconds 2025-02-14 15:46:30,584 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:46:30,584 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29136.07 MB 2025-02-14 15:46:30,584 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40985.07 MB 2025-02-14 15:46:30,584 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11849.00 MB 2025-02-14 15:46:30,584 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42695.92 MB 2025-02-14 15:46:30,584 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47743.76 MB 2025-02-14 15:46:30,584 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5047.84 MB 2025-02-14 15:46:30,584 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41009.85 MB 2025-02-14 15:46:30,846 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 15:46:30,846 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 15:46:30,846 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 15:46:30,846 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:46:30,847 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40985.07 MB 2025-02-14 15:46:30,847 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41085.01 MB 2025-02-14 15:46:30,847 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 99.94 MB 2025-02-14 15:46:30,847 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47743.76 MB 2025-02-14 15:46:30,847 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47743.76 MB 2025-02-14 15:46:30,847 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:46:30,847 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41684.64 MB 2025-02-14 15:46:30,864 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8119, cut from 8121 2025-02-14 15:46:30,864 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 15:46:30,870 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 15:46:30,870 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 15:46:30,871 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:46:30,871 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:46:30,871 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30397.94 MB 2025-02-14 15:46:30,871 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34571.27 MB 2025-02-14 15:46:30,871 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4173.33 MB 2025-02-14 15:46:30,871 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47743.76 MB 2025-02-14 15:46:30,871 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56090.43 MB 2025-02-14 15:46:30,871 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8346.66 MB 2025-02-14 15:46:30,871 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38743.18 MB 2025-02-14 15:46:31,022 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7911] 2025-02-14 15:46:31,023 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:46:31,023 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:46:31,024 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:46:31,024 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 15:46:31,028 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 15:46:31,029 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:46:31,029 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 15:46:31,029 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 15:46:31,030 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:46:31,030 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:46:31,031 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:46:31,031 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:46:31,036 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 15:46:31,037 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:46:31,037 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:46:31,037 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:46:31,037 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:46:31,037 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 15:46:31,038 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:46:31,038 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:46:31,038 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:46:31,038 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 15:46:31,038 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 15:46:31,039 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:46:31,039 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:46:31,042 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:46:31,042 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:46:31,043 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:46:31,043 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:46:31,045 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:46:31,045 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:46:31,049 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:46:31,049 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:47:35,500 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:47:35,500 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:47:35,505 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 15:47:35,507 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:47:35,507 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1122, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 15:47:35,508 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:47:35,508 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1122, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 15:47:52,793 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 15:47:52,793 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 15:47:52,793 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.28 seconds 2025-02-14 15:47:52,793 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:47:52,793 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32840.36 MB 2025-02-14 15:47:52,793 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36811.06 MB 2025-02-14 15:47:52,793 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3970.70 MB 2025-02-14 15:47:52,793 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56090.43 MB 2025-02-14 15:47:52,793 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41846.57 MB 2025-02-14 15:47:52,793 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14243.86 MB 2025-02-14 15:47:52,793 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45709.93 MB 2025-02-14 15:47:52,866 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 15:47:52,866 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 15:47:52,866 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 15:47:52,866 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:47:52,867 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36811.06 MB 2025-02-14 15:47:52,867 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33664.16 MB 2025-02-14 15:47:52,867 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3146.90 MB 2025-02-14 15:47:52,867 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41846.57 MB 2025-02-14 15:47:52,867 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55897.49 MB 2025-02-14 15:47:52,867 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 14050.92 MB 2025-02-14 15:47:52,867 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48917.73 MB 2025-02-14 15:47:54,784 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 15:47:54,784 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 15:47:54,784 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 15:47:54,784 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:47:54,784 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33664.16 MB 2025-02-14 15:47:54,784 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34195.00 MB 2025-02-14 15:47:54,784 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 15:47:54,784 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55897.49 MB 2025-02-14 15:47:54,784 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39290.14 MB 2025-02-14 15:47:54,784 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16607.35 MB 2025-02-14 15:47:54,784 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38174.33 MB 2025-02-14 15:47:54,799 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 15:47:54,799 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 15:47:54,799 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:47:54,799 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:47:54,799 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34195.00 MB 2025-02-14 15:47:54,799 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36084.35 MB 2025-02-14 15:47:54,799 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.35 MB 2025-02-14 15:47:54,799 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39290.14 MB 2025-02-14 15:47:54,799 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41177.58 MB 2025-02-14 15:47:54,799 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 15:47:54,799 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37501.78 MB 2025-02-14 15:47:55,007 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 15:47:55,007 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 15:47:55,007 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 15:47:55,008 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:47:55,008 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36084.35 MB 2025-02-14 15:47:55,008 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38326.21 MB 2025-02-14 15:47:55,008 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 15:47:55,008 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41177.58 MB 2025-02-14 15:47:55,008 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46839.89 MB 2025-02-14 15:47:55,008 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 15:47:55,008 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43870.49 MB 2025-02-14 15:47:55,008 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 15:47:55,008 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 15:47:55,008 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 15:47:55,008 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:47:55,008 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34195.00 MB 2025-02-14 15:47:55,008 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38326.21 MB 2025-02-14 15:47:55,008 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.21 MB 2025-02-14 15:47:55,008 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39290.14 MB 2025-02-14 15:47:55,008 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46839.89 MB 2025-02-14 15:47:55,008 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-14 15:47:55,008 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43870.49 MB 2025-02-14 15:47:55,170 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 15:47:55,170 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 15:47:55,170 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 15:47:55,170 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:47:55,170 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39034.00 MB 2025-02-14 15:47:55,170 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39801.00 MB 2025-02-14 15:47:55,170 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 15:47:55,170 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46839.89 MB 2025-02-14 15:47:55,170 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47255.13 MB 2025-02-14 15:47:55,170 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 15:47:55,170 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40508.79 MB 2025-02-14 15:47:55,189 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 15:47:55,189 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 15:47:55,189 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:47:55,189 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:47:55,189 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40213.89 MB 2025-02-14 15:47:55,189 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40419.61 MB 2025-02-14 15:47:55,189 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.72 MB 2025-02-14 15:47:55,189 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47255.13 MB 2025-02-14 15:47:55,189 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47255.13 MB 2025-02-14 15:47:55,189 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:47:55,189 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40639.00 MB 2025-02-14 15:47:55,190 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 15:47:55,190 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 15:47:55,190 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.68 seconds 2025-02-14 15:47:55,190 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:47:55,190 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28931.22 MB 2025-02-14 15:47:55,190 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40619.78 MB 2025-02-14 15:47:55,190 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11688.55 MB 2025-02-14 15:47:55,190 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56090.43 MB 2025-02-14 15:47:55,190 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47255.13 MB 2025-02-14 15:47:55,190 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8835.30 MB 2025-02-14 15:47:55,190 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40639.00 MB 2025-02-14 15:47:55,455 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 15:47:55,455 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 15:47:55,455 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 15:47:55,455 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:47:55,455 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40619.78 MB 2025-02-14 15:47:55,455 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40719.79 MB 2025-02-14 15:47:55,455 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.01 MB 2025-02-14 15:47:55,455 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47255.13 MB 2025-02-14 15:47:55,455 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47255.13 MB 2025-02-14 15:47:55,455 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:47:55,455 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41319.86 MB 2025-02-14 15:47:55,473 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8125, cut from 8127 2025-02-14 15:47:55,473 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 15:47:55,479 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 15:47:55,479 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 15:47:55,479 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:47:55,479 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:47:55,479 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30193.23 MB 2025-02-14 15:47:55,479 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34368.74 MB 2025-02-14 15:47:55,479 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4175.50 MB 2025-02-14 15:47:55,479 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47255.13 MB 2025-02-14 15:47:55,479 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55608.08 MB 2025-02-14 15:47:55,479 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8352.96 MB 2025-02-14 15:47:55,479 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38544.17 MB 2025-02-14 15:47:55,629 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7917] 2025-02-14 15:47:55,631 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:47:55,631 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:47:55,631 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:47:55,632 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 15:47:55,636 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 15:47:55,637 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:47:55,637 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 15:47:55,637 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 15:47:55,638 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:47:55,638 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:47:55,638 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:47:55,638 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:47:55,644 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 15:47:55,644 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:47:55,644 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:47:55,645 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:47:55,645 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:47:55,645 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 15:47:55,645 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:47:55,645 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:47:55,646 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:47:55,646 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 15:47:55,646 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 15:47:55,646 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:47:55,646 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:47:55,650 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:47:55,650 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:47:55,651 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:47:55,651 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:47:55,652 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:47:55,652 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:47:55,655 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:47:55,656 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:49:00,035 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:49:00,036 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:49:00,040 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 15:49:00,042 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:49:00,042 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1334, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 15:49:00,043 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:49:00,043 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1334, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 15:49:20,485 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 15:49:20,485 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 15:49:20,485 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.43 seconds 2025-02-14 15:49:20,485 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:49:20,485 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34439.98 MB 2025-02-14 15:49:20,485 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39160.93 MB 2025-02-14 15:49:20,485 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4720.95 MB 2025-02-14 15:49:20,485 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57896.08 MB 2025-02-14 15:49:20,485 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49044.00 MB 2025-02-14 15:49:20,485 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8852.08 MB 2025-02-14 15:49:20,485 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47989.13 MB 2025-02-14 15:49:20,564 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 15:49:20,564 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 15:49:20,564 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 15:49:20,564 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:49:20,564 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39160.93 MB 2025-02-14 15:49:20,564 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34888.65 MB 2025-02-14 15:49:20,564 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4272.28 MB 2025-02-14 15:49:20,564 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49044.00 MB 2025-02-14 15:49:20,564 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 58416.17 MB 2025-02-14 15:49:20,564 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9372.17 MB 2025-02-14 15:49:20,564 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 53273.65 MB 2025-02-14 15:49:22,476 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 15:49:22,476 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 15:49:22,477 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 15:49:22,477 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:49:22,477 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34888.65 MB 2025-02-14 15:49:22,477 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35419.49 MB 2025-02-14 15:49:22,477 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 15:49:22,477 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58416.17 MB 2025-02-14 15:49:22,477 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44321.21 MB 2025-02-14 15:49:22,477 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14094.96 MB 2025-02-14 15:49:22,477 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39398.82 MB 2025-02-14 15:49:22,490 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 15:49:22,490 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 15:49:22,491 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:49:22,491 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:49:22,491 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35419.49 MB 2025-02-14 15:49:22,491 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37308.83 MB 2025-02-14 15:49:22,491 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.34 MB 2025-02-14 15:49:22,491 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44321.21 MB 2025-02-14 15:49:22,491 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44321.21 MB 2025-02-14 15:49:22,491 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:49:22,491 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38726.25 MB 2025-02-14 15:49:22,703 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 15:49:22,703 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 15:49:22,703 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 15:49:22,703 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:49:22,703 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37308.83 MB 2025-02-14 15:49:22,703 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39550.68 MB 2025-02-14 15:49:22,703 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 15:49:22,703 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44321.21 MB 2025-02-14 15:49:22,703 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48567.94 MB 2025-02-14 15:49:22,703 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4246.73 MB 2025-02-14 15:49:22,703 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45094.96 MB 2025-02-14 15:49:22,704 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 15:49:22,704 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 15:49:22,704 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 15:49:22,704 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:49:22,704 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35419.49 MB 2025-02-14 15:49:22,704 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39550.68 MB 2025-02-14 15:49:22,704 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.19 MB 2025-02-14 15:49:22,704 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44321.21 MB 2025-02-14 15:49:22,704 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48567.94 MB 2025-02-14 15:49:22,704 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4246.73 MB 2025-02-14 15:49:22,704 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45094.96 MB 2025-02-14 15:49:22,865 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 15:49:22,865 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 15:49:22,865 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 15:49:22,865 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:49:22,865 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40258.47 MB 2025-02-14 15:49:22,865 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41025.47 MB 2025-02-14 15:49:22,865 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 15:49:22,865 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48567.94 MB 2025-02-14 15:49:22,865 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48983.18 MB 2025-02-14 15:49:22,865 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 15:49:22,865 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41733.26 MB 2025-02-14 15:49:22,882 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 15:49:22,882 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 15:49:22,882 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:49:22,882 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:49:22,882 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41438.36 MB 2025-02-14 15:49:22,882 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41644.76 MB 2025-02-14 15:49:22,882 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.40 MB 2025-02-14 15:49:22,882 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48983.18 MB 2025-02-14 15:49:22,882 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48983.18 MB 2025-02-14 15:49:22,882 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:49:22,882 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41863.78 MB 2025-02-14 15:49:22,884 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 15:49:22,884 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 15:49:22,884 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.84 seconds 2025-02-14 15:49:22,884 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:49:22,884 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29792.21 MB 2025-02-14 15:49:22,884 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41845.74 MB 2025-02-14 15:49:22,884 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12053.52 MB 2025-02-14 15:49:22,884 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57896.08 MB 2025-02-14 15:49:22,884 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48983.18 MB 2025-02-14 15:49:22,884 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8912.90 MB 2025-02-14 15:49:22,884 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41863.78 MB 2025-02-14 15:49:23,149 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 15:49:23,149 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 15:49:23,149 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 15:49:23,149 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:49:23,149 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41845.74 MB 2025-02-14 15:49:23,149 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41946.16 MB 2025-02-14 15:49:23,149 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.42 MB 2025-02-14 15:49:23,149 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48983.18 MB 2025-02-14 15:49:23,149 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48983.18 MB 2025-02-14 15:49:23,149 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:49:23,149 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42548.66 MB 2025-02-14 15:49:23,166 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8158, cut from 8160 2025-02-14 15:49:23,167 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 15:49:23,173 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 15:49:23,173 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 15:49:23,173 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:49:23,173 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:49:23,173 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31055.04 MB 2025-02-14 15:49:23,173 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35247.47 MB 2025-02-14 15:49:23,173 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4192.43 MB 2025-02-14 15:49:23,173 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48983.18 MB 2025-02-14 15:49:23,173 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57369.69 MB 2025-02-14 15:49:23,173 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8386.51 MB 2025-02-14 15:49:23,173 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39439.68 MB 2025-02-14 15:49:23,324 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7950] 2025-02-14 15:49:23,325 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:49:23,326 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:49:23,326 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:49:23,326 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 15:49:23,331 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 15:49:23,332 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:49:23,332 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 15:49:23,332 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 15:49:23,333 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:49:23,333 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:49:23,333 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:49:23,333 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:49:23,339 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 15:49:23,339 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:49:23,339 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:49:23,340 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:49:23,340 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:49:23,340 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 15:49:23,340 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:49:23,340 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:49:23,341 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:49:23,341 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 15:49:23,341 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 15:49:23,342 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:49:23,342 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:49:23,345 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:49:23,346 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:49:23,346 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:49:23,346 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:49:23,348 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:49:23,348 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:49:23,352 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:49:23,352 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:50:11,386 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:50:11,386 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:50:11,391 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 15:50:11,392 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:50:11,392 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1529, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 15:50:11,393 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:50:11,393 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1529, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 15:50:35,021 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 15:50:35,022 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 15:50:35,022 - resource_logging.py:150 - __exit__ - DEBUG - Time: 23.62 seconds 2025-02-14 15:50:35,022 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:50:35,022 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35918.93 MB 2025-02-14 15:50:35,022 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41329.97 MB 2025-02-14 15:50:35,022 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5411.05 MB 2025-02-14 15:50:35,022 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61626.91 MB 2025-02-14 15:50:35,022 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51703.19 MB 2025-02-14 15:50:35,022 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9923.72 MB 2025-02-14 15:50:35,022 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50147.44 MB 2025-02-14 15:50:35,115 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 15:50:35,115 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 15:50:35,115 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 15:50:35,115 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:50:35,115 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41329.97 MB 2025-02-14 15:50:35,115 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36022.55 MB 2025-02-14 15:50:35,115 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5307.42 MB 2025-02-14 15:50:35,115 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51703.19 MB 2025-02-14 15:50:35,115 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 62937.63 MB 2025-02-14 15:50:35,115 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 11234.44 MB 2025-02-14 15:50:35,115 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 57343.40 MB 2025-02-14 15:50:37,041 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 15:50:37,041 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 15:50:37,041 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 15:50:37,041 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:50:37,041 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36022.55 MB 2025-02-14 15:50:37,041 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36553.39 MB 2025-02-14 15:50:37,041 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 15:50:37,042 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62937.63 MB 2025-02-14 15:50:37,042 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42117.10 MB 2025-02-14 15:50:37,042 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -20820.53 MB 2025-02-14 15:50:37,042 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40532.72 MB 2025-02-14 15:50:37,057 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 15:50:37,057 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 15:50:37,057 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:50:37,057 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:50:37,057 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36553.39 MB 2025-02-14 15:50:37,057 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38442.51 MB 2025-02-14 15:50:37,057 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.12 MB 2025-02-14 15:50:37,057 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42117.10 MB 2025-02-14 15:50:37,057 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43060.82 MB 2025-02-14 15:50:37,057 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 15:50:37,057 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39859.94 MB 2025-02-14 15:50:37,299 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 15:50:37,299 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 15:50:37,299 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.24 seconds 2025-02-14 15:50:37,299 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:50:37,299 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38442.51 MB 2025-02-14 15:50:37,299 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40684.37 MB 2025-02-14 15:50:37,299 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 15:50:37,299 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43060.82 MB 2025-02-14 15:50:37,299 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48723.13 MB 2025-02-14 15:50:37,299 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 15:50:37,299 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46228.65 MB 2025-02-14 15:50:37,300 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 15:50:37,300 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 15:50:37,300 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 15:50:37,300 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:50:37,300 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36553.39 MB 2025-02-14 15:50:37,300 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40684.37 MB 2025-02-14 15:50:37,300 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4130.98 MB 2025-02-14 15:50:37,300 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42117.10 MB 2025-02-14 15:50:37,300 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48723.13 MB 2025-02-14 15:50:37,300 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6606.03 MB 2025-02-14 15:50:37,300 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46228.65 MB 2025-02-14 15:50:37,467 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 15:50:37,467 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 15:50:37,467 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 15:50:37,467 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:50:37,468 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41392.15 MB 2025-02-14 15:50:37,468 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42159.16 MB 2025-02-14 15:50:37,468 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 15:50:37,468 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48723.13 MB 2025-02-14 15:50:37,468 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49138.37 MB 2025-02-14 15:50:37,468 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 15:50:37,468 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42866.95 MB 2025-02-14 15:50:37,485 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 15:50:37,485 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 15:50:37,485 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:50:37,485 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:50:37,485 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42572.05 MB 2025-02-14 15:50:37,485 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42777.37 MB 2025-02-14 15:50:37,485 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.32 MB 2025-02-14 15:50:37,485 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49138.37 MB 2025-02-14 15:50:37,485 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49138.37 MB 2025-02-14 15:50:37,485 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:50:37,485 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42997.04 MB 2025-02-14 15:50:37,486 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 15:50:37,486 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 15:50:37,486 - resource_logging.py:150 - __exit__ - DEBUG - Time: 26.09 seconds 2025-02-14 15:50:37,486 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:50:37,486 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30591.77 MB 2025-02-14 15:50:37,486 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42977.70 MB 2025-02-14 15:50:37,486 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12385.94 MB 2025-02-14 15:50:37,487 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61626.91 MB 2025-02-14 15:50:37,487 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49138.37 MB 2025-02-14 15:50:37,487 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12488.54 MB 2025-02-14 15:50:37,487 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42997.04 MB 2025-02-14 15:50:37,753 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 15:50:37,753 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 15:50:37,753 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 15:50:37,753 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:50:37,753 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42977.70 MB 2025-02-14 15:50:37,753 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43077.80 MB 2025-02-14 15:50:37,753 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.10 MB 2025-02-14 15:50:37,753 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49138.37 MB 2025-02-14 15:50:37,753 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49138.37 MB 2025-02-14 15:50:37,753 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:50:37,753 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43678.39 MB 2025-02-14 15:50:37,771 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8132, cut from 8134 2025-02-14 15:50:37,771 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-14 15:50:37,777 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 15:50:37,777 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 15:50:37,778 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:50:37,778 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:50:37,778 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31853.95 MB 2025-02-14 15:50:37,778 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36033.57 MB 2025-02-14 15:50:37,778 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4179.62 MB 2025-02-14 15:50:37,778 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49138.37 MB 2025-02-14 15:50:37,778 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57497.62 MB 2025-02-14 15:50:37,778 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8359.25 MB 2025-02-14 15:50:37,778 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40213.20 MB 2025-02-14 15:50:37,940 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7924] 2025-02-14 15:50:37,942 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:50:37,942 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:50:37,943 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:50:37,943 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 15:50:37,947 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 15:50:37,948 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:50:37,948 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 15:50:37,948 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-14 15:50:37,949 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:50:37,949 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:50:37,950 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:50:37,950 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:50:37,956 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 15:50:37,956 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:50:37,956 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:50:37,957 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:50:37,957 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:50:37,957 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 15:50:37,957 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:50:37,957 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:50:37,958 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:50:37,958 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 15:50:37,958 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 15:50:37,958 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:50:37,958 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:50:37,964 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:50:37,964 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:50:37,965 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:50:37,965 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:50:37,966 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:50:37,966 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:50:37,972 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:50:37,972 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:51:28,401 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:51:28,401 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:51:28,408 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 15:51:28,410 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:51:28,410 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1187, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 15:51:28,413 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:51:28,413 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1187, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 15:51:46,837 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 15:51:46,837 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 15:51:46,837 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.41 seconds 2025-02-14 15:51:46,837 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:51:46,837 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33658.27 MB 2025-02-14 15:51:46,837 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37859.00 MB 2025-02-14 15:51:46,837 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4200.73 MB 2025-02-14 15:51:46,837 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61876.47 MB 2025-02-14 15:51:46,837 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42259.71 MB 2025-02-14 15:51:46,837 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19616.76 MB 2025-02-14 15:51:46,837 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46754.33 MB 2025-02-14 15:51:46,932 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 15:51:46,932 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 15:51:46,932 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 15:51:46,932 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:51:46,932 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37859.00 MB 2025-02-14 15:51:46,932 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34367.05 MB 2025-02-14 15:51:46,932 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3491.95 MB 2025-02-14 15:51:46,932 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42259.71 MB 2025-02-14 15:51:46,932 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57824.77 MB 2025-02-14 15:51:46,932 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 15565.06 MB 2025-02-14 15:51:46,932 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50418.00 MB 2025-02-14 15:51:48,867 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 15:51:48,867 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 15:51:48,867 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 15:51:48,867 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:51:48,867 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34367.05 MB 2025-02-14 15:51:48,867 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34897.90 MB 2025-02-14 15:51:48,867 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 15:51:48,867 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57824.77 MB 2025-02-14 15:51:48,867 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39472.59 MB 2025-02-14 15:51:48,867 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18352.18 MB 2025-02-14 15:51:48,867 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38877.23 MB 2025-02-14 15:51:48,881 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 15:51:48,881 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 15:51:48,882 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:51:48,882 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:51:48,882 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34897.90 MB 2025-02-14 15:51:48,882 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36787.07 MB 2025-02-14 15:51:48,882 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.17 MB 2025-02-14 15:51:48,882 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39472.59 MB 2025-02-14 15:51:48,882 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41360.03 MB 2025-02-14 15:51:48,882 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 15:51:48,882 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38204.49 MB 2025-02-14 15:51:49,100 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 15:51:49,100 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 15:51:49,100 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 15:51:49,100 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:51:49,100 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36787.07 MB 2025-02-14 15:51:49,100 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39028.92 MB 2025-02-14 15:51:49,100 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 15:51:49,100 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41360.03 MB 2025-02-14 15:51:49,100 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47022.34 MB 2025-02-14 15:51:49,100 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 15:51:49,100 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44573.20 MB 2025-02-14 15:51:49,101 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 15:51:49,101 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 15:51:49,101 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 15:51:49,101 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:51:49,101 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34897.90 MB 2025-02-14 15:51:49,101 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39028.92 MB 2025-02-14 15:51:49,101 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.03 MB 2025-02-14 15:51:49,101 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39472.59 MB 2025-02-14 15:51:49,101 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47022.34 MB 2025-02-14 15:51:49,101 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-14 15:51:49,101 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44573.20 MB 2025-02-14 15:51:49,268 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 15:51:49,268 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 15:51:49,268 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 15:51:49,268 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:51:49,268 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39736.71 MB 2025-02-14 15:51:49,268 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40503.71 MB 2025-02-14 15:51:49,268 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 15:51:49,268 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47022.34 MB 2025-02-14 15:51:49,268 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47437.58 MB 2025-02-14 15:51:49,268 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 15:51:49,268 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41211.50 MB 2025-02-14 15:51:49,286 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 15:51:49,286 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 15:51:49,286 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:51:49,286 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:51:49,286 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40916.60 MB 2025-02-14 15:51:49,286 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41121.96 MB 2025-02-14 15:51:49,286 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.36 MB 2025-02-14 15:51:49,286 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47437.58 MB 2025-02-14 15:51:49,286 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47437.58 MB 2025-02-14 15:51:49,286 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:51:49,286 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41344.65 MB 2025-02-14 15:51:49,287 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 15:51:49,287 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 15:51:49,287 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.87 seconds 2025-02-14 15:51:49,287 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:51:49,287 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29522.67 MB 2025-02-14 15:51:49,287 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41321.88 MB 2025-02-14 15:51:49,287 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11799.21 MB 2025-02-14 15:51:49,287 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61876.47 MB 2025-02-14 15:51:49,287 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47437.58 MB 2025-02-14 15:51:49,287 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14438.89 MB 2025-02-14 15:51:49,287 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41344.65 MB 2025-02-14 15:51:49,554 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 15:51:49,555 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 15:51:49,555 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 15:51:49,555 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:51:49,555 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41321.88 MB 2025-02-14 15:51:49,555 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41421.77 MB 2025-02-14 15:51:49,555 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 99.89 MB 2025-02-14 15:51:49,555 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47437.58 MB 2025-02-14 15:51:49,555 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47437.58 MB 2025-02-14 15:51:49,555 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:51:49,555 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42021.10 MB 2025-02-14 15:51:49,572 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8115, cut from 8117 2025-02-14 15:51:49,573 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 15:51:49,579 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 15:51:49,579 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 15:51:49,579 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:51:49,579 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:51:49,579 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30784.44 MB 2025-02-14 15:51:49,579 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34955.67 MB 2025-02-14 15:51:49,579 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4171.24 MB 2025-02-14 15:51:49,579 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47437.58 MB 2025-02-14 15:51:49,579 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55780.05 MB 2025-02-14 15:51:49,579 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8342.47 MB 2025-02-14 15:51:49,579 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39125.53 MB 2025-02-14 15:51:49,742 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7907] 2025-02-14 15:51:49,744 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:51:49,744 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:51:49,745 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:51:49,745 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 15:51:49,749 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 15:51:49,750 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:51:49,751 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 15:51:49,751 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 15:51:49,752 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:51:49,752 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:51:49,752 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:51:49,752 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:51:49,758 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 15:51:49,759 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:51:49,759 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:51:49,759 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:51:49,759 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:51:49,759 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 15:51:49,760 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:51:49,760 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:51:49,760 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:51:49,760 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 15:51:49,760 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 15:51:49,761 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:51:49,761 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:51:49,771 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:51:49,771 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:51:49,773 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:51:49,773 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:51:49,775 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:51:49,775 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:51:49,780 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:51:49,780 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:51:58,264 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:51:58,264 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:51:58,269 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 15:51:58,270 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:51:58,270 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1136, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 15:51:58,271 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:51:58,271 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1136, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 15:52:16,054 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 15:52:16,055 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 15:52:16,055 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.77 seconds 2025-02-14 15:52:16,055 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:52:16,055 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33424.56 MB 2025-02-14 15:52:16,055 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37444.80 MB 2025-02-14 15:52:16,055 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4020.24 MB 2025-02-14 15:52:16,055 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60280.54 MB 2025-02-14 15:52:16,055 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42198.89 MB 2025-02-14 15:52:16,055 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18081.64 MB 2025-02-14 15:52:16,055 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46294.12 MB 2025-02-14 15:52:16,182 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 15:52:16,182 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 15:52:16,182 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 15:52:16,182 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:52:16,182 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37444.80 MB 2025-02-14 15:52:16,182 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34223.58 MB 2025-02-14 15:52:16,182 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3221.22 MB 2025-02-14 15:52:16,182 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42198.89 MB 2025-02-14 15:52:16,182 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56727.96 MB 2025-02-14 15:52:16,182 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 14529.07 MB 2025-02-14 15:52:16,182 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49706.66 MB 2025-02-14 15:52:18,168 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 15:52:18,169 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 15:52:18,169 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.98 seconds 2025-02-14 15:52:18,169 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:52:18,169 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34223.58 MB 2025-02-14 15:52:18,169 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34754.42 MB 2025-02-14 15:52:18,169 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 15:52:18,169 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56727.96 MB 2025-02-14 15:52:18,169 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39594.23 MB 2025-02-14 15:52:18,169 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17133.73 MB 2025-02-14 15:52:18,169 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38733.76 MB 2025-02-14 15:52:18,183 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 15:52:18,183 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 15:52:18,183 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:52:18,183 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:52:18,183 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34754.42 MB 2025-02-14 15:52:18,183 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36643.64 MB 2025-02-14 15:52:18,183 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.22 MB 2025-02-14 15:52:18,183 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39594.23 MB 2025-02-14 15:52:18,183 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41481.67 MB 2025-02-14 15:52:18,183 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 15:52:18,183 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38061.07 MB 2025-02-14 15:52:18,397 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 15:52:18,397 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 15:52:18,397 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 15:52:18,397 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:52:18,397 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36643.64 MB 2025-02-14 15:52:18,397 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38885.50 MB 2025-02-14 15:52:18,397 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 15:52:18,397 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41481.67 MB 2025-02-14 15:52:18,397 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47143.98 MB 2025-02-14 15:52:18,397 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 15:52:18,397 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44429.78 MB 2025-02-14 15:52:18,398 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 15:52:18,398 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 15:52:18,398 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 15:52:18,398 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:52:18,398 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34754.42 MB 2025-02-14 15:52:18,398 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38885.50 MB 2025-02-14 15:52:18,398 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.07 MB 2025-02-14 15:52:18,398 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39594.23 MB 2025-02-14 15:52:18,398 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47143.98 MB 2025-02-14 15:52:18,398 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-14 15:52:18,398 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44429.78 MB 2025-02-14 15:52:18,563 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 15:52:18,564 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 15:52:18,564 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 15:52:18,564 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:52:18,564 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39593.29 MB 2025-02-14 15:52:18,564 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40360.29 MB 2025-02-14 15:52:18,564 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 15:52:18,564 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47143.98 MB 2025-02-14 15:52:18,564 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47559.21 MB 2025-02-14 15:52:18,564 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 15:52:18,564 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41068.08 MB 2025-02-14 15:52:18,581 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 15:52:18,581 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 15:52:18,581 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:52:18,581 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:52:18,581 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40773.18 MB 2025-02-14 15:52:18,581 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40978.84 MB 2025-02-14 15:52:18,581 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.66 MB 2025-02-14 15:52:18,581 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47559.21 MB 2025-02-14 15:52:18,581 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47559.21 MB 2025-02-14 15:52:18,581 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:52:18,581 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41202.68 MB 2025-02-14 15:52:18,582 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 15:52:18,582 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 15:52:18,582 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.31 seconds 2025-02-14 15:52:18,582 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:52:18,582 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29466.64 MB 2025-02-14 15:52:18,582 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41179.22 MB 2025-02-14 15:52:18,582 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11712.58 MB 2025-02-14 15:52:18,582 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60280.54 MB 2025-02-14 15:52:18,582 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47559.21 MB 2025-02-14 15:52:18,582 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12721.32 MB 2025-02-14 15:52:18,582 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41202.68 MB 2025-02-14 15:52:18,851 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 15:52:18,851 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 15:52:18,851 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.27 seconds 2025-02-14 15:52:18,851 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:52:18,851 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41179.22 MB 2025-02-14 15:52:18,851 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41279.35 MB 2025-02-14 15:52:18,851 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.12 MB 2025-02-14 15:52:18,851 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47559.21 MB 2025-02-14 15:52:18,851 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47559.21 MB 2025-02-14 15:52:18,851 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:52:18,851 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41880.08 MB 2025-02-14 15:52:18,869 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8134, cut from 8136 2025-02-14 15:52:18,870 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 15:52:18,876 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 15:52:18,876 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 15:52:18,876 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:52:18,876 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:52:18,876 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30728.88 MB 2025-02-14 15:52:18,876 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34909.00 MB 2025-02-14 15:52:18,876 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4180.12 MB 2025-02-14 15:52:18,876 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47559.21 MB 2025-02-14 15:52:18,876 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55920.56 MB 2025-02-14 15:52:18,876 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8361.35 MB 2025-02-14 15:52:18,876 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39088.62 MB 2025-02-14 15:52:19,039 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7926] 2025-02-14 15:52:19,041 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:52:19,041 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:52:19,042 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:52:19,042 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 15:52:19,047 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 15:52:19,048 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:52:19,048 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 15:52:19,048 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 15:52:19,049 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:52:19,049 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:52:19,049 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:52:19,049 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:52:19,055 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 15:52:19,056 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:52:19,056 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:52:19,056 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:52:19,056 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:52:19,056 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 15:52:19,057 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:52:19,057 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:52:19,057 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:52:19,057 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 15:52:19,057 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 15:52:19,058 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:52:19,058 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:52:19,063 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:52:19,063 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:52:19,065 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:52:19,065 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:52:19,066 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:52:19,066 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:52:19,071 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:52:19,071 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:53:13,589 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:53:13,589 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:53:13,594 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 15:53:13,595 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:53:13,595 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 187, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 15:53:13,596 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:53:13,596 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 187, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 15:53:16,517 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 15:53:16,517 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 15:53:16,517 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.92 seconds 2025-02-14 15:53:16,517 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:53:16,517 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26932.83 MB 2025-02-14 15:53:16,517 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27594.61 MB 2025-02-14 15:53:16,517 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 661.78 MB 2025-02-14 15:53:16,517 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60542.68 MB 2025-02-14 15:53:16,517 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 29943.14 MB 2025-02-14 15:53:16,517 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30599.54 MB 2025-02-14 15:53:16,517 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36405.01 MB 2025-02-14 15:53:16,534 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 15:53:16,534 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 15:53:16,534 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:53:16,534 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:53:16,534 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27594.61 MB 2025-02-14 15:53:16,534 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27916.85 MB 2025-02-14 15:53:16,534 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 322.24 MB 2025-02-14 15:53:16,534 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 29943.14 MB 2025-02-14 15:53:16,534 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31920.75 MB 2025-02-14 15:53:16,534 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1977.61 MB 2025-02-14 15:53:16,534 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30262.74 MB 2025-02-14 15:53:17,440 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 15:53:17,440 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 15:53:17,440 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.90 seconds 2025-02-14 15:53:17,440 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:53:17,440 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27916.85 MB 2025-02-14 15:53:17,440 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28165.02 MB 2025-02-14 15:53:17,440 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 248.17 MB 2025-02-14 15:53:17,440 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31920.75 MB 2025-02-14 15:53:17,440 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31258.05 MB 2025-02-14 15:53:17,440 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -662.70 MB 2025-02-14 15:53:17,440 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32088.33 MB 2025-02-14 15:53:17,448 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 15:53:17,449 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 15:53:17,449 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:53:17,449 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:53:17,449 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28164.96 MB 2025-02-14 15:53:17,449 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29048.10 MB 2025-02-14 15:53:17,449 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 883.14 MB 2025-02-14 15:53:17,449 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31258.05 MB 2025-02-14 15:53:17,449 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31258.05 MB 2025-02-14 15:53:17,449 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:53:17,449 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29710.75 MB 2025-02-14 15:53:17,553 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 15:53:17,553 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 15:53:17,553 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 15:53:17,553 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:53:17,553 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29048.10 MB 2025-02-14 15:53:17,553 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30096.20 MB 2025-02-14 15:53:17,553 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1048.10 MB 2025-02-14 15:53:17,553 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31258.05 MB 2025-02-14 15:53:17,553 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33915.14 MB 2025-02-14 15:53:17,553 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2657.09 MB 2025-02-14 15:53:17,553 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32688.12 MB 2025-02-14 15:53:17,553 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 15:53:17,554 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 15:53:17,554 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 15:53:17,554 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:53:17,554 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28164.96 MB 2025-02-14 15:53:17,554 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30096.20 MB 2025-02-14 15:53:17,554 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1931.25 MB 2025-02-14 15:53:17,554 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31258.05 MB 2025-02-14 15:53:17,554 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33915.14 MB 2025-02-14 15:53:17,554 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2657.09 MB 2025-02-14 15:53:17,554 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32688.12 MB 2025-02-14 15:53:17,634 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 15:53:17,634 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 15:53:17,634 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 15:53:17,634 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:53:17,634 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30427.09 MB 2025-02-14 15:53:17,634 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30785.67 MB 2025-02-14 15:53:17,634 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 358.57 MB 2025-02-14 15:53:17,634 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33915.14 MB 2025-02-14 15:53:17,634 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34105.98 MB 2025-02-14 15:53:17,634 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 190.84 MB 2025-02-14 15:53:17,634 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31124.23 MB 2025-02-14 15:53:17,644 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 15:53:17,644 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 15:53:17,644 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:53:17,644 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:53:17,644 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30978.70 MB 2025-02-14 15:53:17,644 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31179.73 MB 2025-02-14 15:53:17,644 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 201.03 MB 2025-02-14 15:53:17,644 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34105.98 MB 2025-02-14 15:53:17,644 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34105.98 MB 2025-02-14 15:53:17,644 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:53:17,644 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31209.71 MB 2025-02-14 15:53:17,645 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 15:53:17,645 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 15:53:17,645 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.05 seconds 2025-02-14 15:53:17,645 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:53:17,645 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26281.31 MB 2025-02-14 15:53:17,646 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31380.80 MB 2025-02-14 15:53:17,646 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5099.49 MB 2025-02-14 15:53:17,646 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60542.68 MB 2025-02-14 15:53:17,646 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34105.98 MB 2025-02-14 15:53:17,646 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26436.70 MB 2025-02-14 15:53:17,646 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31380.80 MB 2025-02-14 15:53:17,909 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 15:53:17,909 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 15:53:17,909 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 15:53:17,909 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:53:17,909 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31380.80 MB 2025-02-14 15:53:17,909 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31481.26 MB 2025-02-14 15:53:17,909 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.47 MB 2025-02-14 15:53:17,909 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34105.98 MB 2025-02-14 15:53:17,909 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34105.98 MB 2025-02-14 15:53:17,909 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:53:17,909 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32084.06 MB 2025-02-14 15:53:17,935 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 15:53:17,936 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-14 15:53:17,943 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 15:53:17,943 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 15:53:17,943 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 15:53:17,943 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:53:17,943 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26978.76 MB 2025-02-14 15:53:17,943 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31173.25 MB 2025-02-14 15:53:17,943 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4194.49 MB 2025-02-14 15:53:17,943 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34105.98 MB 2025-02-14 15:53:17,943 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44595.94 MB 2025-02-14 15:53:17,943 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 15:53:17,943 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35367.55 MB 2025-02-14 15:53:18,109 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 15:53:18,110 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:53:18,111 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:53:18,111 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:53:18,111 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 15:53:18,116 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 15:53:18,117 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:53:18,117 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 15:53:18,117 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-14 15:53:18,118 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:53:18,118 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:53:18,119 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:53:18,119 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:53:18,125 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 15:53:18,125 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:53:18,125 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:53:18,126 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:53:18,126 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:53:18,126 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 15:53:18,126 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:53:18,126 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:53:18,127 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:53:18,127 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 15:53:18,127 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 15:53:18,127 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:53:18,127 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:53:18,132 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:53:18,132 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:53:18,134 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:53:18,134 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:53:18,135 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:53:18,135 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:53:18,141 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:53:18,141 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:53:34,000 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:53:34,001 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:53:34,006 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 15:53:34,007 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:53:34,007 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1178, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 15:53:34,008 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:53:34,008 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1178, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 15:53:52,212 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 15:53:52,212 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 15:53:52,212 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.20 seconds 2025-02-14 15:53:52,212 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:53:52,212 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33960.54 MB 2025-02-14 15:53:52,212 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38129.68 MB 2025-02-14 15:53:52,212 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4169.14 MB 2025-02-14 15:53:52,212 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49339.70 MB 2025-02-14 15:53:52,212 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44723.86 MB 2025-02-14 15:53:52,212 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4615.83 MB 2025-02-14 15:53:52,212 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47055.79 MB 2025-02-14 15:53:52,287 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 15:53:52,287 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 15:53:52,288 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 15:53:52,288 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:53:52,288 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38129.68 MB 2025-02-14 15:53:52,288 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34685.25 MB 2025-02-14 15:53:52,288 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3444.43 MB 2025-02-14 15:53:52,288 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44723.86 MB 2025-02-14 15:53:52,288 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56497.27 MB 2025-02-14 15:53:52,288 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 11773.41 MB 2025-02-14 15:53:52,288 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50663.37 MB 2025-02-14 15:53:54,214 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 15:53:54,214 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 15:53:54,214 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 15:53:54,214 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:53:54,214 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34685.25 MB 2025-02-14 15:53:54,214 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35216.09 MB 2025-02-14 15:53:54,214 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 15:53:54,214 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56497.27 MB 2025-02-14 15:53:54,214 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41970.30 MB 2025-02-14 15:53:54,214 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14526.97 MB 2025-02-14 15:53:54,214 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39195.42 MB 2025-02-14 15:53:54,227 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 15:53:54,227 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 15:53:54,227 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:53:54,227 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:53:54,227 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35216.09 MB 2025-02-14 15:53:54,227 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37105.40 MB 2025-02-14 15:53:54,227 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.31 MB 2025-02-14 15:53:54,227 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41970.30 MB 2025-02-14 15:53:54,227 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41970.30 MB 2025-02-14 15:53:54,227 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:53:54,227 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38522.83 MB 2025-02-14 15:53:54,438 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 15:53:54,438 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 15:53:54,438 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 15:53:54,438 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:53:54,438 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37105.40 MB 2025-02-14 15:53:54,438 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39347.26 MB 2025-02-14 15:53:54,438 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 15:53:54,438 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41970.30 MB 2025-02-14 15:53:54,438 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48104.47 MB 2025-02-14 15:53:54,438 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 15:53:54,438 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44891.54 MB 2025-02-14 15:53:54,438 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 15:53:54,438 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 15:53:54,439 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 15:53:54,439 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:53:54,439 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35216.09 MB 2025-02-14 15:53:54,439 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39347.26 MB 2025-02-14 15:53:54,439 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.17 MB 2025-02-14 15:53:54,439 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41970.30 MB 2025-02-14 15:53:54,439 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48104.47 MB 2025-02-14 15:53:54,439 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 15:53:54,439 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44891.54 MB 2025-02-14 15:53:54,602 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 15:53:54,602 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 15:53:54,602 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 15:53:54,602 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:53:54,602 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40055.05 MB 2025-02-14 15:53:54,602 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40822.05 MB 2025-02-14 15:53:54,602 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 15:53:54,602 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48104.47 MB 2025-02-14 15:53:54,602 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48519.71 MB 2025-02-14 15:53:54,602 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 15:53:54,602 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41529.84 MB 2025-02-14 15:53:54,619 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 15:53:54,619 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 15:53:54,619 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:53:54,619 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:53:54,619 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41234.94 MB 2025-02-14 15:53:54,619 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41441.78 MB 2025-02-14 15:53:54,619 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.85 MB 2025-02-14 15:53:54,619 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48519.71 MB 2025-02-14 15:53:54,619 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48519.71 MB 2025-02-14 15:53:54,619 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:53:54,619 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41659.25 MB 2025-02-14 15:53:54,620 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 15:53:54,620 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 15:53:54,620 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.61 seconds 2025-02-14 15:53:54,620 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:53:54,620 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29856.29 MB 2025-02-14 15:53:54,620 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41642.64 MB 2025-02-14 15:53:54,620 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11786.34 MB 2025-02-14 15:53:54,620 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49339.70 MB 2025-02-14 15:53:54,620 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48519.71 MB 2025-02-14 15:53:54,621 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -819.99 MB 2025-02-14 15:53:54,621 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41659.25 MB 2025-02-14 15:53:54,883 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 15:53:54,883 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 15:53:54,883 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 15:53:54,883 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:53:54,883 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41642.64 MB 2025-02-14 15:53:54,883 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41742.99 MB 2025-02-14 15:53:54,883 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.36 MB 2025-02-14 15:53:54,883 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48519.71 MB 2025-02-14 15:53:54,883 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48519.71 MB 2025-02-14 15:53:54,883 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:53:54,883 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42345.13 MB 2025-02-14 15:53:54,901 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8153, cut from 8155 2025-02-14 15:53:54,901 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 15:53:54,907 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 15:53:54,908 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 15:53:54,908 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:53:54,908 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:53:54,908 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31118.99 MB 2025-02-14 15:53:54,908 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35309.10 MB 2025-02-14 15:53:54,908 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4190.11 MB 2025-02-14 15:53:54,908 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48519.71 MB 2025-02-14 15:53:54,908 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56899.93 MB 2025-02-14 15:53:54,908 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8380.22 MB 2025-02-14 15:53:54,908 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39499.21 MB 2025-02-14 15:53:55,075 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7945] 2025-02-14 15:53:55,076 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:53:55,076 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:53:55,077 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:53:55,077 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 15:53:55,082 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 15:53:55,083 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:53:55,083 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 15:53:55,083 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 15:53:55,084 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:53:55,084 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:53:55,084 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:53:55,084 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:53:55,090 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 15:53:55,090 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:53:55,091 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:53:55,091 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:53:55,091 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:53:55,091 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 15:53:55,091 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:53:55,092 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:53:55,092 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:53:55,092 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 15:53:55,092 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 15:53:55,093 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:53:55,093 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:53:55,099 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:53:55,099 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:53:55,101 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:53:55,101 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:53:55,103 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:53:55,103 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:53:55,109 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:53:55,109 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:54:50,069 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:54:50,069 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:54:50,077 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 15:54:50,080 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:54:50,080 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 261, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 15:54:50,081 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:54:50,081 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 261, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 15:54:54,228 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 15:54:54,229 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 15:54:54,229 - resource_logging.py:150 - __exit__ - DEBUG - Time: 4.14 seconds 2025-02-14 15:54:54,229 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:54:54,229 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27692.40 MB 2025-02-14 15:54:54,229 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28616.06 MB 2025-02-14 15:54:54,229 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 923.66 MB 2025-02-14 15:54:54,229 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61765.32 MB 2025-02-14 15:54:54,229 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34854.67 MB 2025-02-14 15:54:54,229 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -26910.65 MB 2025-02-14 15:54:54,229 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37616.75 MB 2025-02-14 15:54:54,253 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 15:54:54,253 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 15:54:54,253 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:54:54,253 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:54:54,253 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28616.06 MB 2025-02-14 15:54:54,253 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28916.03 MB 2025-02-14 15:54:54,254 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 299.96 MB 2025-02-14 15:54:54,254 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34854.67 MB 2025-02-14 15:54:54,254 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34854.67 MB 2025-02-14 15:54:54,254 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:54:54,254 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32004.81 MB 2025-02-14 15:54:55,408 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 15:54:55,408 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 15:54:55,408 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.15 seconds 2025-02-14 15:54:55,408 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:54:55,408 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28916.03 MB 2025-02-14 15:54:55,408 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29234.53 MB 2025-02-14 15:54:55,408 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 318.50 MB 2025-02-14 15:54:55,408 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34854.67 MB 2025-02-14 15:54:55,408 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34382.81 MB 2025-02-14 15:54:55,408 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -471.86 MB 2025-02-14 15:54:55,408 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33171.40 MB 2025-02-14 15:54:55,418 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 15:54:55,418 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 15:54:55,418 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:54:55,418 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:54:55,418 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29234.53 MB 2025-02-14 15:54:55,418 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30367.98 MB 2025-02-14 15:54:55,418 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1133.45 MB 2025-02-14 15:54:55,418 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34382.81 MB 2025-02-14 15:54:55,418 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34382.81 MB 2025-02-14 15:54:55,418 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:54:55,418 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31218.44 MB 2025-02-14 15:54:55,550 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 15:54:55,550 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 15:54:55,550 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 15:54:55,550 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:54:55,550 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30367.98 MB 2025-02-14 15:54:55,550 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31713.12 MB 2025-02-14 15:54:55,550 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1345.14 MB 2025-02-14 15:54:55,550 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34382.81 MB 2025-02-14 15:54:55,550 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36930.85 MB 2025-02-14 15:54:55,550 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2548.04 MB 2025-02-14 15:54:55,550 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35039.66 MB 2025-02-14 15:54:55,551 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 15:54:55,551 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 15:54:55,551 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-14 15:54:55,551 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:54:55,551 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29234.53 MB 2025-02-14 15:54:55,551 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31713.12 MB 2025-02-14 15:54:55,551 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2478.58 MB 2025-02-14 15:54:55,551 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34382.81 MB 2025-02-14 15:54:55,551 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36930.85 MB 2025-02-14 15:54:55,551 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2548.04 MB 2025-02-14 15:54:55,551 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35039.66 MB 2025-02-14 15:54:55,649 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 15:54:55,649 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 15:54:55,649 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 15:54:55,649 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:54:55,649 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32137.79 MB 2025-02-14 15:54:55,649 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32597.99 MB 2025-02-14 15:54:55,649 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 460.20 MB 2025-02-14 15:54:55,649 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36930.85 MB 2025-02-14 15:54:55,649 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37178.31 MB 2025-02-14 15:54:55,649 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 247.46 MB 2025-02-14 15:54:55,650 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33022.66 MB 2025-02-14 15:54:55,661 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 15:54:55,661 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 15:54:55,661 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:54:55,661 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:54:55,661 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32845.73 MB 2025-02-14 15:54:55,661 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33052.20 MB 2025-02-14 15:54:55,661 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.47 MB 2025-02-14 15:54:55,661 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37178.31 MB 2025-02-14 15:54:55,661 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37178.31 MB 2025-02-14 15:54:55,661 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:54:55,661 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33116.66 MB 2025-02-14 15:54:55,662 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 15:54:55,662 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 15:54:55,662 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.58 seconds 2025-02-14 15:54:55,662 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:54:55,662 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26783.05 MB 2025-02-14 15:54:55,662 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33253.27 MB 2025-02-14 15:54:55,662 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6470.21 MB 2025-02-14 15:54:55,662 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61765.32 MB 2025-02-14 15:54:55,662 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37178.31 MB 2025-02-14 15:54:55,662 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24587.01 MB 2025-02-14 15:54:55,662 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33253.27 MB 2025-02-14 15:54:55,923 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 15:54:55,923 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 15:54:55,924 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 15:54:55,924 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:54:55,924 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33253.27 MB 2025-02-14 15:54:55,924 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33353.73 MB 2025-02-14 15:54:55,924 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.47 MB 2025-02-14 15:54:55,924 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37178.31 MB 2025-02-14 15:54:55,924 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37178.31 MB 2025-02-14 15:54:55,924 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:54:55,924 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33956.53 MB 2025-02-14 15:54:55,942 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 15:54:55,942 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 15:54:55,949 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 15:54:55,949 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 15:54:55,949 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:54:55,949 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:54:55,949 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33353.73 MB 2025-02-14 15:54:55,949 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31815.69 MB 2025-02-14 15:54:55,949 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1538.04 MB 2025-02-14 15:54:55,949 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37178.31 MB 2025-02-14 15:54:55,949 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45569.02 MB 2025-02-14 15:54:55,949 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 15:54:55,949 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36010.00 MB 2025-02-14 15:54:56,111 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 15:54:56,112 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:54:56,112 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:54:56,113 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:54:56,113 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 15:54:56,118 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 15:54:56,119 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:54:56,119 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 15:54:56,119 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 15:54:56,120 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:54:56,120 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:54:56,120 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:54:56,120 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:54:56,126 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 15:54:56,127 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:54:56,127 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:54:56,127 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:54:56,127 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:54:56,127 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 15:54:56,128 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:54:56,128 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:54:56,128 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:54:56,128 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 15:54:56,128 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 15:54:56,129 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:54:56,129 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:54:56,134 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:54:56,134 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:54:56,136 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:54:56,136 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:54:56,138 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:54:56,138 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:54:56,144 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:54:56,144 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:55:12,891 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:55:12,891 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:55:12,896 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 15:55:12,897 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:55:12,897 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1161, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 15:55:12,898 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:55:12,898 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1161, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 15:55:30,832 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 15:55:30,832 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 15:55:30,832 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.93 seconds 2025-02-14 15:55:30,832 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:55:30,832 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34085.00 MB 2025-02-14 15:55:30,832 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38193.71 MB 2025-02-14 15:55:30,832 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4108.71 MB 2025-02-14 15:55:30,832 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50556.04 MB 2025-02-14 15:55:30,832 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42809.16 MB 2025-02-14 15:55:30,832 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7746.88 MB 2025-02-14 15:55:30,832 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47181.05 MB 2025-02-14 15:55:30,910 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 15:55:30,910 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 15:55:30,910 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 15:55:30,910 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:55:30,910 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38193.71 MB 2025-02-14 15:55:30,910 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34839.78 MB 2025-02-14 15:55:30,910 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3353.93 MB 2025-02-14 15:55:30,910 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42809.16 MB 2025-02-14 15:55:30,910 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57810.09 MB 2025-02-14 15:55:30,910 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 15000.93 MB 2025-02-14 15:55:30,910 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50604.93 MB 2025-02-14 15:55:32,827 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 15:55:32,827 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 15:55:32,827 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 15:55:32,827 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:55:32,827 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34839.78 MB 2025-02-14 15:55:32,827 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35370.63 MB 2025-02-14 15:55:32,827 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 15:55:32,827 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57810.09 MB 2025-02-14 15:55:32,828 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40114.32 MB 2025-02-14 15:55:32,828 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17695.77 MB 2025-02-14 15:55:32,828 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39349.96 MB 2025-02-14 15:55:32,841 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 15:55:32,841 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 15:55:32,841 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:55:32,841 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:55:32,841 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35370.63 MB 2025-02-14 15:55:32,841 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37259.98 MB 2025-02-14 15:55:32,841 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.35 MB 2025-02-14 15:55:32,841 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40114.32 MB 2025-02-14 15:55:32,841 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42001.76 MB 2025-02-14 15:55:32,841 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 15:55:32,841 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38677.41 MB 2025-02-14 15:55:33,053 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 15:55:33,053 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 15:55:33,053 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 15:55:33,053 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:55:33,053 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37259.98 MB 2025-02-14 15:55:33,053 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39501.84 MB 2025-02-14 15:55:33,053 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 15:55:33,053 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42001.76 MB 2025-02-14 15:55:33,053 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48135.93 MB 2025-02-14 15:55:33,053 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 15:55:33,053 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45046.12 MB 2025-02-14 15:55:33,054 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 15:55:33,054 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 15:55:33,054 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 15:55:33,054 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:55:33,054 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35370.63 MB 2025-02-14 15:55:33,054 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39501.84 MB 2025-02-14 15:55:33,054 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.21 MB 2025-02-14 15:55:33,054 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40114.32 MB 2025-02-14 15:55:33,054 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48135.93 MB 2025-02-14 15:55:33,054 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-14 15:55:33,054 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45046.12 MB 2025-02-14 15:55:33,218 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 15:55:33,218 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 15:55:33,218 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 15:55:33,218 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:55:33,218 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40209.63 MB 2025-02-14 15:55:33,218 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40976.63 MB 2025-02-14 15:55:33,218 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 15:55:33,218 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48135.93 MB 2025-02-14 15:55:33,218 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48551.17 MB 2025-02-14 15:55:33,218 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 15:55:33,218 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41684.42 MB 2025-02-14 15:55:33,235 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 15:55:33,235 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 15:55:33,235 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:55:33,235 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:55:33,235 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41389.52 MB 2025-02-14 15:55:33,235 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41594.91 MB 2025-02-14 15:55:33,235 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.39 MB 2025-02-14 15:55:33,235 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48551.17 MB 2025-02-14 15:55:33,235 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48551.17 MB 2025-02-14 15:55:33,235 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:55:33,235 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41820.34 MB 2025-02-14 15:55:33,236 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 15:55:33,236 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 15:55:33,236 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.34 seconds 2025-02-14 15:55:33,236 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:55:33,236 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30039.98 MB 2025-02-14 15:55:33,236 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41795.37 MB 2025-02-14 15:55:33,236 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11755.38 MB 2025-02-14 15:55:33,236 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50556.04 MB 2025-02-14 15:55:33,236 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48551.17 MB 2025-02-14 15:55:33,236 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -2004.88 MB 2025-02-14 15:55:33,236 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41820.34 MB 2025-02-14 15:55:33,499 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 15:55:33,499 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 15:55:33,499 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 15:55:33,499 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:55:33,499 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41795.37 MB 2025-02-14 15:55:33,499 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41895.52 MB 2025-02-14 15:55:33,499 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.16 MB 2025-02-14 15:55:33,499 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48551.17 MB 2025-02-14 15:55:33,499 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48551.17 MB 2025-02-14 15:55:33,499 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:55:33,499 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42496.48 MB 2025-02-14 15:55:33,517 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8137, cut from 8139 2025-02-14 15:55:33,518 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 15:55:33,524 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 15:55:33,524 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 15:55:33,524 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:55:33,524 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:55:33,524 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31302.29 MB 2025-02-14 15:55:33,524 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35484.01 MB 2025-02-14 15:55:33,524 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4181.72 MB 2025-02-14 15:55:33,524 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48551.17 MB 2025-02-14 15:55:33,524 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 59005.47 MB 2025-02-14 15:55:33,524 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10454.30 MB 2025-02-14 15:55:33,524 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39665.73 MB 2025-02-14 15:55:33,687 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7929] 2025-02-14 15:55:33,689 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:55:33,689 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:55:33,690 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:55:33,690 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 15:55:33,694 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 15:55:33,695 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:55:33,695 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 15:55:33,696 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 15:55:33,696 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:55:33,696 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:55:33,697 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:55:33,697 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:55:33,703 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 15:55:33,703 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:55:33,703 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:55:33,704 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:55:33,704 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:55:33,704 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 15:55:33,704 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:55:33,704 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:55:33,705 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:55:33,705 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 15:55:33,705 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 15:55:33,705 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:55:33,705 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:55:33,711 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:55:33,711 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:55:33,713 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:55:33,713 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:55:33,715 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:55:33,715 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:55:33,721 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:55:33,721 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:55:40,928 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:55:40,928 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:55:40,933 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 15:55:40,935 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:55:40,935 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 359, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 15:55:40,936 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:55:40,936 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 359, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 15:55:46,543 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 15:55:46,543 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 15:55:46,543 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.60 seconds 2025-02-14 15:55:46,543 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:55:46,543 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28618.60 MB 2025-02-14 15:55:46,543 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29889.47 MB 2025-02-14 15:55:46,543 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1270.87 MB 2025-02-14 15:55:46,543 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64114.13 MB 2025-02-14 15:55:46,543 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34531.70 MB 2025-02-14 15:55:46,543 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29582.43 MB 2025-02-14 15:55:46,543 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38770.25 MB 2025-02-14 15:55:46,566 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 15:55:46,566 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 15:55:46,566 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:55:46,566 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:55:46,566 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29889.47 MB 2025-02-14 15:55:46,566 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30119.24 MB 2025-02-14 15:55:46,566 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 229.77 MB 2025-02-14 15:55:46,566 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34531.70 MB 2025-02-14 15:55:46,566 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37205.57 MB 2025-02-14 15:55:46,566 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2673.87 MB 2025-02-14 15:55:46,566 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34186.21 MB 2025-02-14 15:55:48,026 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 15:55:48,027 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 15:55:48,027 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.46 seconds 2025-02-14 15:55:48,027 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:55:48,027 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30119.24 MB 2025-02-14 15:55:48,027 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30522.68 MB 2025-02-14 15:55:48,027 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 403.44 MB 2025-02-14 15:55:48,027 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37205.57 MB 2025-02-14 15:55:48,027 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33915.14 MB 2025-02-14 15:55:48,027 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3290.43 MB 2025-02-14 15:55:48,027 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34460.59 MB 2025-02-14 15:55:48,038 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 15:55:48,038 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 15:55:48,038 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:55:48,038 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:55:48,038 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30522.68 MB 2025-02-14 15:55:48,038 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31959.05 MB 2025-02-14 15:55:48,038 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1436.37 MB 2025-02-14 15:55:48,038 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33915.14 MB 2025-02-14 15:55:48,038 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36066.82 MB 2025-02-14 15:55:48,038 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2151.68 MB 2025-02-14 15:55:48,038 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33036.30 MB 2025-02-14 15:55:48,205 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 15:55:48,205 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 15:55:48,205 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 15:55:48,205 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:55:48,205 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31959.05 MB 2025-02-14 15:55:48,205 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33662.88 MB 2025-02-14 15:55:48,205 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1703.83 MB 2025-02-14 15:55:48,205 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36066.82 MB 2025-02-14 15:55:48,205 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40370.18 MB 2025-02-14 15:55:48,205 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4303.36 MB 2025-02-14 15:55:48,205 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37876.52 MB 2025-02-14 15:55:48,206 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 15:55:48,206 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 15:55:48,206 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-14 15:55:48,206 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:55:48,206 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30522.68 MB 2025-02-14 15:55:48,206 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33662.88 MB 2025-02-14 15:55:48,206 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3140.19 MB 2025-02-14 15:55:48,206 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33915.14 MB 2025-02-14 15:55:48,206 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40370.18 MB 2025-02-14 15:55:48,206 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6455.03 MB 2025-02-14 15:55:48,206 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37876.52 MB 2025-02-14 15:55:48,336 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 15:55:48,336 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 15:55:48,336 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 15:55:48,336 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:55:48,336 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34200.80 MB 2025-02-14 15:55:48,336 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34783.72 MB 2025-02-14 15:55:48,336 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 582.92 MB 2025-02-14 15:55:48,336 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40370.18 MB 2025-02-14 15:55:48,336 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40682.65 MB 2025-02-14 15:55:48,336 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 312.48 MB 2025-02-14 15:55:48,336 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35321.64 MB 2025-02-14 15:55:48,350 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 15:55:48,350 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 15:55:48,350 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:55:48,350 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:55:48,350 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35097.52 MB 2025-02-14 15:55:48,350 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35304.20 MB 2025-02-14 15:55:48,350 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.68 MB 2025-02-14 15:55:48,350 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40682.65 MB 2025-02-14 15:55:48,350 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40682.65 MB 2025-02-14 15:55:48,350 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:55:48,350 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35408.92 MB 2025-02-14 15:55:48,351 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 15:55:48,351 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 15:55:48,351 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.41 seconds 2025-02-14 15:55:48,351 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:55:48,351 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27367.81 MB 2025-02-14 15:55:48,351 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35505.10 MB 2025-02-14 15:55:48,351 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8137.28 MB 2025-02-14 15:55:48,351 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64114.13 MB 2025-02-14 15:55:48,351 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40682.65 MB 2025-02-14 15:55:48,351 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23431.48 MB 2025-02-14 15:55:48,351 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35505.10 MB 2025-02-14 15:55:48,615 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 15:55:48,615 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 15:55:48,615 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 15:55:48,615 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:55:48,615 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35505.10 MB 2025-02-14 15:55:48,615 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35605.48 MB 2025-02-14 15:55:48,615 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.38 MB 2025-02-14 15:55:48,615 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40682.65 MB 2025-02-14 15:55:48,615 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40682.65 MB 2025-02-14 15:55:48,615 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:55:48,615 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36207.76 MB 2025-02-14 15:55:48,642 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8155, cut from 8157 2025-02-14 15:55:48,642 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 15:55:48,648 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 15:55:48,648 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 15:55:48,648 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 15:55:48,648 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:55:48,648 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35605.48 MB 2025-02-14 15:55:48,648 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32566.60 MB 2025-02-14 15:55:48,648 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3038.88 MB 2025-02-14 15:55:48,648 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40682.65 MB 2025-02-14 15:55:48,648 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51164.22 MB 2025-02-14 15:55:48,648 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10481.57 MB 2025-02-14 15:55:48,649 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36756.98 MB 2025-02-14 15:55:48,811 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7947] 2025-02-14 15:55:48,813 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:55:48,813 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:55:48,814 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:55:48,814 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 15:55:48,818 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 15:55:48,819 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:55:48,819 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 15:55:48,819 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 15:55:48,820 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:55:48,820 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:55:48,821 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:55:48,821 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:55:48,827 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 15:55:48,827 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:55:48,827 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:55:48,828 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:55:48,828 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:55:48,828 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 15:55:48,828 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:55:48,828 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:55:48,829 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:55:48,829 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 15:55:48,829 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 15:55:48,829 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:55:48,829 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:55:48,834 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:55:48,834 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:55:48,835 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:55:48,835 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:55:48,836 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:55:48,836 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:55:48,842 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:55:48,842 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:57:13,549 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:57:13,549 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:57:13,554 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 15:57:13,555 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:57:13,555 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 132, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 15:57:13,556 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:57:13,556 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 132, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 15:57:15,620 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 15:57:15,620 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 15:57:15,620 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.06 seconds 2025-02-14 15:57:15,620 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:57:15,620 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27159.21 MB 2025-02-14 15:57:15,620 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27626.35 MB 2025-02-14 15:57:15,620 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 467.14 MB 2025-02-14 15:57:15,620 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56394.51 MB 2025-02-14 15:57:15,620 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31073.50 MB 2025-02-14 15:57:15,620 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25321.01 MB 2025-02-14 15:57:15,620 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36631.39 MB 2025-02-14 15:57:15,630 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 15:57:15,630 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 15:57:15,630 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:57:15,630 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:57:15,631 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27626.35 MB 2025-02-14 15:57:15,631 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27684.13 MB 2025-02-14 15:57:15,631 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 57.78 MB 2025-02-14 15:57:15,631 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31073.50 MB 2025-02-14 15:57:15,631 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31073.50 MB 2025-02-14 15:57:15,631 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:57:15,631 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29157.54 MB 2025-02-14 15:57:16,147 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 15:57:16,147 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 15:57:16,147 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.51 seconds 2025-02-14 15:57:16,147 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:57:16,147 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27684.13 MB 2025-02-14 15:57:16,147 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27827.45 MB 2025-02-14 15:57:16,147 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 143.33 MB 2025-02-14 15:57:16,147 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31073.50 MB 2025-02-14 15:57:16,147 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31073.50 MB 2025-02-14 15:57:16,147 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:57:16,147 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31770.67 MB 2025-02-14 15:57:16,153 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 15:57:16,153 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 15:57:16,153 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 15:57:16,153 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:57:16,153 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27827.39 MB 2025-02-14 15:57:16,153 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28337.44 MB 2025-02-14 15:57:16,153 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 510.05 MB 2025-02-14 15:57:16,153 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31073.50 MB 2025-02-14 15:57:16,153 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31073.50 MB 2025-02-14 15:57:16,153 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:57:16,153 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28720.15 MB 2025-02-14 15:57:16,260 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 15:57:16,260 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 15:57:16,260 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 15:57:16,260 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:57:16,260 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28337.44 MB 2025-02-14 15:57:16,260 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28957.73 MB 2025-02-14 15:57:16,260 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 620.29 MB 2025-02-14 15:57:16,260 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31073.50 MB 2025-02-14 15:57:16,260 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31585.21 MB 2025-02-14 15:57:16,260 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 511.71 MB 2025-02-14 15:57:16,260 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30441.79 MB 2025-02-14 15:57:16,261 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 15:57:16,261 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 15:57:16,261 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 15:57:16,261 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:57:16,261 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27827.39 MB 2025-02-14 15:57:16,261 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28957.73 MB 2025-02-14 15:57:16,261 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1130.34 MB 2025-02-14 15:57:16,261 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31073.50 MB 2025-02-14 15:57:16,261 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31585.21 MB 2025-02-14 15:57:16,261 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 511.71 MB 2025-02-14 15:57:16,261 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30441.79 MB 2025-02-14 15:57:16,311 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 15:57:16,311 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 15:57:16,311 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-14 15:57:16,311 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:57:16,311 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29233.77 MB 2025-02-14 15:57:16,311 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29494.40 MB 2025-02-14 15:57:16,311 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 260.63 MB 2025-02-14 15:57:16,311 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31585.21 MB 2025-02-14 15:57:16,311 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31748.78 MB 2025-02-14 15:57:16,311 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 163.58 MB 2025-02-14 15:57:16,311 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29685.50 MB 2025-02-14 15:57:16,316 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 15:57:16,316 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 15:57:16,316 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 15:57:16,316 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:57:16,316 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29658.97 MB 2025-02-14 15:57:16,316 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29858.61 MB 2025-02-14 15:57:16,316 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 199.64 MB 2025-02-14 15:57:16,316 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31748.78 MB 2025-02-14 15:57:16,317 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31748.78 MB 2025-02-14 15:57:16,317 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:57:16,317 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29858.61 MB 2025-02-14 15:57:16,318 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 15:57:16,318 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 15:57:16,318 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.76 seconds 2025-02-14 15:57:16,318 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:57:16,318 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26699.31 MB 2025-02-14 15:57:16,318 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30059.41 MB 2025-02-14 15:57:16,318 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3360.10 MB 2025-02-14 15:57:16,318 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56394.51 MB 2025-02-14 15:57:16,318 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31748.78 MB 2025-02-14 15:57:16,318 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24645.73 MB 2025-02-14 15:57:16,318 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30059.41 MB 2025-02-14 15:57:16,579 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 15:57:16,579 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 15:57:16,579 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 15:57:16,579 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:57:16,579 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27086.89 MB 2025-02-14 15:57:16,579 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27187.22 MB 2025-02-14 15:57:16,580 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.33 MB 2025-02-14 15:57:16,580 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31748.78 MB 2025-02-14 15:57:16,580 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31748.78 MB 2025-02-14 15:57:16,580 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:57:16,580 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27789.21 MB 2025-02-14 15:57:16,597 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8151, cut from 8153 2025-02-14 15:57:16,598 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 15:57:16,604 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 15:57:16,604 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 15:57:16,604 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:57:16,604 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:57:16,604 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27187.22 MB 2025-02-14 15:57:16,604 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31376.06 MB 2025-02-14 15:57:16,604 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4188.84 MB 2025-02-14 15:57:16,604 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31748.78 MB 2025-02-14 15:57:16,604 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42224.06 MB 2025-02-14 15:57:16,604 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10475.27 MB 2025-02-14 15:57:16,604 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35564.39 MB 2025-02-14 15:57:16,755 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7943] 2025-02-14 15:57:16,756 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:57:16,756 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:57:16,757 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:57:16,757 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 15:57:16,761 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 15:57:16,762 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:57:16,762 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 15:57:16,762 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 15:57:16,763 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:57:16,763 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:57:16,764 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:57:16,764 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:57:16,769 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 15:57:16,770 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:57:16,770 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:57:16,770 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:57:16,770 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:57:16,770 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 15:57:16,771 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:57:16,771 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:57:16,771 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:57:16,771 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 15:57:16,771 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 15:57:16,772 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:57:16,772 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:57:16,775 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:57:16,775 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:57:16,776 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:57:16,776 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:57:16,777 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:57:16,777 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:57:16,781 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:57:16,781 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:58:11,162 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:58:11,162 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:58:11,168 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 15:58:11,169 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:58:11,169 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2076, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 15:58:11,170 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:58:11,170 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2076, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 15:58:43,048 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 15:58:43,048 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 15:58:43,048 - resource_logging.py:150 - __exit__ - DEBUG - Time: 31.87 seconds 2025-02-14 15:58:43,048 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:58:43,048 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40826.87 MB 2025-02-14 15:58:43,048 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48173.72 MB 2025-02-14 15:58:43,048 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 7346.85 MB 2025-02-14 15:58:43,048 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51134.86 MB 2025-02-14 15:58:43,048 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55190.75 MB 2025-02-14 15:58:43,048 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4055.89 MB 2025-02-14 15:58:43,048 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 57093.01 MB 2025-02-14 15:58:43,196 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 15:58:43,196 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 15:58:43,196 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 15:58:43,196 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:58:43,196 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 48173.72 MB 2025-02-14 15:58:43,196 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39962.59 MB 2025-02-14 15:58:43,196 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -8211.13 MB 2025-02-14 15:58:43,196 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55190.75 MB 2025-02-14 15:58:43,196 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 79064.73 MB 2025-02-14 15:58:43,196 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 23873.98 MB 2025-02-14 15:58:43,196 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 69974.18 MB 2025-02-14 15:58:45,137 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 15:58:45,138 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 15:58:45,138 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-14 15:58:45,138 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:58:45,138 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39962.59 MB 2025-02-14 15:58:45,138 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40493.43 MB 2025-02-14 15:58:45,138 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 15:58:45,138 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 79064.73 MB 2025-02-14 15:58:45,138 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45067.80 MB 2025-02-14 15:58:45,138 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33996.93 MB 2025-02-14 15:58:45,138 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44473.80 MB 2025-02-14 15:58:45,151 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 15:58:45,152 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 15:58:45,152 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:58:45,152 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:58:45,152 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40493.43 MB 2025-02-14 15:58:45,152 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42382.79 MB 2025-02-14 15:58:45,152 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.35 MB 2025-02-14 15:58:45,152 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45067.80 MB 2025-02-14 15:58:45,152 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46955.23 MB 2025-02-14 15:58:45,152 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 15:58:45,152 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43800.21 MB 2025-02-14 15:58:45,373 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 15:58:45,373 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 15:58:45,373 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 15:58:45,373 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:58:45,373 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42382.79 MB 2025-02-14 15:58:45,373 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44624.64 MB 2025-02-14 15:58:45,373 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 15:58:45,373 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46955.23 MB 2025-02-14 15:58:45,373 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53089.40 MB 2025-02-14 15:58:45,373 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 15:58:45,373 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50168.92 MB 2025-02-14 15:58:45,374 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 15:58:45,374 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 15:58:45,374 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 15:58:45,374 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:58:45,374 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40493.43 MB 2025-02-14 15:58:45,374 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44624.64 MB 2025-02-14 15:58:45,374 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.21 MB 2025-02-14 15:58:45,374 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45067.80 MB 2025-02-14 15:58:45,374 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53089.40 MB 2025-02-14 15:58:45,374 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-14 15:58:45,374 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50168.92 MB 2025-02-14 15:58:45,544 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 15:58:45,544 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 15:58:45,544 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 15:58:45,544 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:58:45,544 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45332.43 MB 2025-02-14 15:58:45,544 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46099.43 MB 2025-02-14 15:58:45,544 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 15:58:45,544 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53089.40 MB 2025-02-14 15:58:45,544 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53504.64 MB 2025-02-14 15:58:45,544 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 15:58:45,544 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46807.22 MB 2025-02-14 15:58:45,562 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 15:58:45,562 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 15:58:45,562 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:58:45,562 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:58:45,562 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46512.32 MB 2025-02-14 15:58:45,562 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46718.44 MB 2025-02-14 15:58:45,562 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.12 MB 2025-02-14 15:58:45,562 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53504.64 MB 2025-02-14 15:58:45,562 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53504.64 MB 2025-02-14 15:58:45,562 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:58:45,562 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46944.12 MB 2025-02-14 15:58:45,563 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 15:58:45,563 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 15:58:45,563 - resource_logging.py:150 - __exit__ - DEBUG - Time: 34.39 seconds 2025-02-14 15:58:45,563 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:58:45,563 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33593.92 MB 2025-02-14 15:58:45,563 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46919.36 MB 2025-02-14 15:58:45,563 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13325.44 MB 2025-02-14 15:58:45,563 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47573.89 MB 2025-02-14 15:58:45,563 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53504.64 MB 2025-02-14 15:58:45,563 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5930.75 MB 2025-02-14 15:58:45,563 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46944.12 MB 2025-02-14 15:58:45,829 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 15:58:45,829 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 15:58:45,829 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 15:58:45,829 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:58:45,829 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46919.36 MB 2025-02-14 15:58:45,829 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47019.76 MB 2025-02-14 15:58:45,829 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.39 MB 2025-02-14 15:58:45,829 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53504.64 MB 2025-02-14 15:58:45,829 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53504.64 MB 2025-02-14 15:58:45,829 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:58:45,829 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47622.11 MB 2025-02-14 15:58:45,847 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8156, cut from 8158 2025-02-14 15:58:45,848 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-14 15:58:45,854 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 15:58:45,854 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 15:58:45,854 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:58:45,854 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:58:45,854 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34856.69 MB 2025-02-14 15:58:45,854 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39048.90 MB 2025-02-14 15:58:45,854 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4192.21 MB 2025-02-14 15:58:45,854 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53504.64 MB 2025-02-14 15:58:45,854 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 61889.05 MB 2025-02-14 15:58:45,854 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8384.41 MB 2025-02-14 15:58:45,854 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43239.79 MB 2025-02-14 15:58:46,022 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7948] 2025-02-14 15:58:46,023 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:58:46,023 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:58:46,024 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:58:46,024 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 15:58:46,029 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 15:58:46,030 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:58:46,030 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 15:58:46,030 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-14 15:58:46,031 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:58:46,031 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:58:46,031 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:58:46,031 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:58:46,037 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 15:58:46,038 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:58:46,038 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:58:46,038 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:58:46,038 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:58:46,038 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 15:58:46,039 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:58:46,039 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:58:46,039 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:58:46,039 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 15:58:46,040 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 15:58:46,040 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:58:46,040 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:58:46,045 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:58:46,046 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:58:46,047 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:58:46,047 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:58:46,049 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:58:46,049 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:58:46,055 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:58:46,055 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:58:59,233 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:58:59,233 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:58:59,238 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 15:58:59,239 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:58:59,239 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1256, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 15:58:59,240 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:58:59,240 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1256, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 15:59:18,744 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 15:59:18,744 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 15:59:18,744 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.50 seconds 2025-02-14 15:59:18,744 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:59:18,744 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35234.09 MB 2025-02-14 15:59:18,744 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39680.05 MB 2025-02-14 15:59:18,744 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4445.96 MB 2025-02-14 15:59:18,744 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 67360.52 MB 2025-02-14 15:59:18,744 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51373.93 MB 2025-02-14 15:59:18,744 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15986.59 MB 2025-02-14 15:59:18,744 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48555.83 MB 2025-02-14 15:59:18,820 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 15:59:18,820 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 15:59:18,820 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 15:59:18,820 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:59:18,820 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39680.05 MB 2025-02-14 15:59:18,820 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35820.78 MB 2025-02-14 15:59:18,820 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3859.27 MB 2025-02-14 15:59:18,820 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51373.93 MB 2025-02-14 15:59:18,820 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 60148.42 MB 2025-02-14 15:59:18,820 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8774.48 MB 2025-02-14 15:59:18,820 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52901.83 MB 2025-02-14 15:59:20,781 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 15:59:20,781 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 15:59:20,782 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.96 seconds 2025-02-14 15:59:20,782 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:59:20,782 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35820.78 MB 2025-02-14 15:59:20,782 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36351.62 MB 2025-02-14 15:59:20,782 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 15:59:20,782 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60148.42 MB 2025-02-14 15:59:20,782 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46927.97 MB 2025-02-14 15:59:20,782 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13220.45 MB 2025-02-14 15:59:20,782 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40330.95 MB 2025-02-14 15:59:20,795 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 15:59:20,795 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 15:59:20,795 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 15:59:20,796 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:59:20,796 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36351.62 MB 2025-02-14 15:59:20,796 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38240.97 MB 2025-02-14 15:59:20,796 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.35 MB 2025-02-14 15:59:20,796 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46927.97 MB 2025-02-14 15:59:20,796 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46927.97 MB 2025-02-14 15:59:20,796 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:59:20,796 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39658.40 MB 2025-02-14 15:59:21,010 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 15:59:21,010 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 15:59:21,010 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 15:59:21,010 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:59:21,010 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38240.97 MB 2025-02-14 15:59:21,010 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40482.83 MB 2025-02-14 15:59:21,010 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 15:59:21,010 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46927.97 MB 2025-02-14 15:59:21,010 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49287.27 MB 2025-02-14 15:59:21,010 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2359.30 MB 2025-02-14 15:59:21,010 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46027.11 MB 2025-02-14 15:59:21,011 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 15:59:21,011 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 15:59:21,011 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 15:59:21,011 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:59:21,011 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36351.62 MB 2025-02-14 15:59:21,011 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40482.83 MB 2025-02-14 15:59:21,011 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.21 MB 2025-02-14 15:59:21,011 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46927.97 MB 2025-02-14 15:59:21,011 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49287.27 MB 2025-02-14 15:59:21,011 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2359.30 MB 2025-02-14 15:59:21,011 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46027.11 MB 2025-02-14 15:59:21,178 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 15:59:21,178 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 15:59:21,178 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 15:59:21,178 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:59:21,178 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41190.62 MB 2025-02-14 15:59:21,178 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41957.62 MB 2025-02-14 15:59:21,178 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 15:59:21,178 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49287.27 MB 2025-02-14 15:59:21,178 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49702.50 MB 2025-02-14 15:59:21,178 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 15:59:21,178 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42665.41 MB 2025-02-14 15:59:21,196 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 15:59:21,196 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 15:59:21,196 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:59:21,196 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:59:21,196 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42370.51 MB 2025-02-14 15:59:21,196 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42576.11 MB 2025-02-14 15:59:21,196 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.60 MB 2025-02-14 15:59:21,196 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49702.50 MB 2025-02-14 15:59:21,196 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49702.50 MB 2025-02-14 15:59:21,196 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:59:21,196 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42798.20 MB 2025-02-14 15:59:21,197 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 15:59:21,197 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 15:59:21,197 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.96 seconds 2025-02-14 15:59:21,197 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:59:21,197 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30858.08 MB 2025-02-14 15:59:21,197 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42776.17 MB 2025-02-14 15:59:21,197 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11918.09 MB 2025-02-14 15:59:21,197 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 67360.52 MB 2025-02-14 15:59:21,197 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49702.50 MB 2025-02-14 15:59:21,197 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17658.02 MB 2025-02-14 15:59:21,197 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42798.20 MB 2025-02-14 15:59:21,463 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 15:59:21,463 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 15:59:21,463 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 15:59:21,463 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:59:21,463 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42776.17 MB 2025-02-14 15:59:21,463 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42876.14 MB 2025-02-14 15:59:21,463 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 99.96 MB 2025-02-14 15:59:21,463 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49702.50 MB 2025-02-14 15:59:21,463 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49702.50 MB 2025-02-14 15:59:21,463 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 15:59:21,463 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43475.91 MB 2025-02-14 15:59:21,481 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8121, cut from 8123 2025-02-14 15:59:21,481 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 15:59:21,487 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 15:59:21,487 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 15:59:21,487 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 15:59:21,487 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 15:59:21,487 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32120.00 MB 2025-02-14 15:59:21,487 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36293.45 MB 2025-02-14 15:59:21,487 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4173.45 MB 2025-02-14 15:59:21,487 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49702.50 MB 2025-02-14 15:59:21,487 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53875.83 MB 2025-02-14 15:59:21,487 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4173.33 MB 2025-02-14 15:59:21,487 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40466.78 MB 2025-02-14 15:59:21,650 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7913] 2025-02-14 15:59:21,652 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:59:21,652 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:59:21,653 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:59:21,653 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 15:59:21,657 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 15:59:21,658 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:59:21,658 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 15:59:21,658 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 15:59:21,659 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:59:21,659 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:59:21,660 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:59:21,660 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:59:21,666 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 15:59:21,666 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:59:21,666 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:59:21,667 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:59:21,667 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:59:21,667 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 15:59:21,667 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:59:21,667 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:59:21,668 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:59:21,668 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 15:59:21,668 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 15:59:21,668 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:59:21,669 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 15:59:21,674 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:59:21,674 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:59:21,676 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:59:21,676 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:59:21,678 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:59:21,678 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 15:59:21,685 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 15:59:21,685 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:00:38,743 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:00:38,744 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:00:38,751 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 16:00:38,754 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:00:38,754 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 375, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 16:00:38,756 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:00:38,756 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 375, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 16:00:44,525 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 16:00:44,525 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 16:00:44,525 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.76 seconds 2025-02-14 16:00:44,525 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:00:44,525 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29217.18 MB 2025-02-14 16:00:44,525 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30544.67 MB 2025-02-14 16:00:44,525 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1327.50 MB 2025-02-14 16:00:44,525 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59468.94 MB 2025-02-14 16:00:44,525 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34129.05 MB 2025-02-14 16:00:44,525 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -25339.89 MB 2025-02-14 16:00:44,525 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39368.83 MB 2025-02-14 16:00:44,552 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 16:00:44,552 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 16:00:44,552 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 16:00:44,552 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:00:44,552 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30544.67 MB 2025-02-14 16:00:44,552 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30844.04 MB 2025-02-14 16:00:44,552 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 299.37 MB 2025-02-14 16:00:44,552 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34129.05 MB 2025-02-14 16:00:44,552 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37559.99 MB 2025-02-14 16:00:44,552 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3430.94 MB 2025-02-14 16:00:44,552 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35150.41 MB 2025-02-14 16:00:46,121 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 16:00:46,121 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 16:00:46,122 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.57 seconds 2025-02-14 16:00:46,122 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:00:46,122 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30844.04 MB 2025-02-14 16:00:46,122 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31276.68 MB 2025-02-14 16:00:46,122 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 432.64 MB 2025-02-14 16:00:46,122 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37559.99 MB 2025-02-14 16:00:46,122 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33573.31 MB 2025-02-14 16:00:46,122 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3986.69 MB 2025-02-14 16:00:46,122 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35270.32 MB 2025-02-14 16:00:46,134 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 16:00:46,134 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 16:00:46,134 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 16:00:46,134 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:00:46,134 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31276.68 MB 2025-02-14 16:00:46,134 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32817.82 MB 2025-02-14 16:00:46,134 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1541.14 MB 2025-02-14 16:00:46,134 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33573.31 MB 2025-02-14 16:00:46,134 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35882.27 MB 2025-02-14 16:00:46,134 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2308.96 MB 2025-02-14 16:00:46,134 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33973.55 MB 2025-02-14 16:00:46,301 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 16:00:46,301 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 16:00:46,301 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 16:00:46,301 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:00:46,301 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32817.82 MB 2025-02-14 16:00:46,301 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34645.47 MB 2025-02-14 16:00:46,301 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1827.65 MB 2025-02-14 16:00:46,301 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35882.27 MB 2025-02-14 16:00:46,301 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41081.11 MB 2025-02-14 16:00:46,301 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5198.84 MB 2025-02-14 16:00:46,301 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39166.80 MB 2025-02-14 16:00:46,302 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 16:00:46,302 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 16:00:46,302 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.18 seconds 2025-02-14 16:00:46,302 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:00:46,302 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31276.68 MB 2025-02-14 16:00:46,302 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34645.47 MB 2025-02-14 16:00:46,302 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3368.79 MB 2025-02-14 16:00:46,302 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33573.31 MB 2025-02-14 16:00:46,302 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41081.11 MB 2025-02-14 16:00:46,302 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7507.80 MB 2025-02-14 16:00:46,302 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39166.80 MB 2025-02-14 16:00:46,433 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 16:00:46,433 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 16:00:46,433 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 16:00:46,433 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:00:46,433 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35222.32 MB 2025-02-14 16:00:46,433 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35848.08 MB 2025-02-14 16:00:46,433 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 625.76 MB 2025-02-14 16:00:46,433 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41081.11 MB 2025-02-14 16:00:46,433 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41418.75 MB 2025-02-14 16:00:46,433 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 337.64 MB 2025-02-14 16:00:46,433 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36424.93 MB 2025-02-14 16:00:46,447 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 16:00:46,447 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 16:00:46,447 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 16:00:46,447 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:00:46,447 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36184.59 MB 2025-02-14 16:00:46,447 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36390.30 MB 2025-02-14 16:00:46,447 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.72 MB 2025-02-14 16:00:46,447 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41418.75 MB 2025-02-14 16:00:46,447 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41418.75 MB 2025-02-14 16:00:46,447 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:00:46,447 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36515.71 MB 2025-02-14 16:00:46,448 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 16:00:46,448 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 16:00:46,448 - resource_logging.py:150 - __exit__ - DEBUG - Time: 7.69 seconds 2025-02-14 16:00:46,448 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:00:46,448 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27910.65 MB 2025-02-14 16:00:46,448 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36591.38 MB 2025-02-14 16:00:46,448 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8680.73 MB 2025-02-14 16:00:46,448 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59468.94 MB 2025-02-14 16:00:46,448 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41418.75 MB 2025-02-14 16:00:46,448 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18050.19 MB 2025-02-14 16:00:46,448 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36591.38 MB 2025-02-14 16:00:46,709 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 16:00:46,709 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 16:00:46,709 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 16:00:46,709 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:00:46,709 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36591.38 MB 2025-02-14 16:00:46,709 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36691.84 MB 2025-02-14 16:00:46,709 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.47 MB 2025-02-14 16:00:46,709 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41418.75 MB 2025-02-14 16:00:46,709 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41418.75 MB 2025-02-14 16:00:46,709 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:00:46,709 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37294.64 MB 2025-02-14 16:00:46,733 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 16:00:46,734 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 16:00:46,740 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 16:00:46,740 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 16:00:46,740 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 16:00:46,740 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:00:46,740 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36691.84 MB 2025-02-14 16:00:46,740 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33172.25 MB 2025-02-14 16:00:46,740 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3519.59 MB 2025-02-14 16:00:46,740 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41418.75 MB 2025-02-14 16:00:46,740 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51908.71 MB 2025-02-14 16:00:46,740 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10489.95 MB 2025-02-14 16:00:46,740 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37366.56 MB 2025-02-14 16:00:46,895 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 16:00:46,897 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:00:46,897 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:00:46,898 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:00:46,898 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 16:00:46,902 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 16:00:46,903 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:00:46,903 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 16:00:46,903 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 16:00:46,904 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:00:46,904 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:00:46,905 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:00:46,905 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:00:46,910 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 16:00:46,911 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:00:46,911 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:00:46,911 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:00:46,911 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:00:46,911 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 16:00:46,912 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:00:46,912 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:00:46,912 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:00:46,912 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 16:00:46,912 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 16:00:46,913 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:00:46,913 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:00:46,917 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:00:46,917 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:00:46,919 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:00:46,919 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:00:46,920 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:00:46,920 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:00:46,926 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:00:46,926 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:01:32,933 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:01:32,934 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:01:32,947 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 16:01:32,950 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:01:32,950 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1732, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 16:01:32,952 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:01:32,952 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1732, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 16:01:59,641 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 16:01:59,642 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 16:01:59,642 - resource_logging.py:150 - __exit__ - DEBUG - Time: 26.68 seconds 2025-02-14 16:01:59,642 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:01:59,642 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38794.75 MB 2025-02-14 16:01:59,642 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44924.73 MB 2025-02-14 16:01:59,642 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6129.98 MB 2025-02-14 16:01:59,642 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60595.11 MB 2025-02-14 16:01:59,642 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54121.20 MB 2025-02-14 16:01:59,642 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6473.91 MB 2025-02-14 16:01:59,642 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 53928.43 MB 2025-02-14 16:01:59,743 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 16:01:59,743 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 16:01:59,743 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 16:01:59,743 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:01:59,743 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44924.73 MB 2025-02-14 16:01:59,743 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38539.17 MB 2025-02-14 16:01:59,743 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6385.56 MB 2025-02-14 16:01:59,743 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54121.20 MB 2025-02-14 16:01:59,743 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 69275.22 MB 2025-02-14 16:01:59,743 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 15154.02 MB 2025-02-14 16:01:59,744 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 62960.01 MB 2025-02-14 16:02:01,666 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 16:02:01,666 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 16:02:01,666 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 16:02:01,666 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:02:01,666 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38539.17 MB 2025-02-14 16:02:01,666 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39070.01 MB 2025-02-14 16:02:01,666 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 16:02:01,666 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 69275.22 MB 2025-02-14 16:02:01,666 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47991.23 MB 2025-02-14 16:02:01,666 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21284.00 MB 2025-02-14 16:02:01,666 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43049.35 MB 2025-02-14 16:02:01,679 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 16:02:01,680 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 16:02:01,680 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 16:02:01,680 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:02:01,680 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39070.01 MB 2025-02-14 16:02:01,680 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40959.37 MB 2025-02-14 16:02:01,680 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.35 MB 2025-02-14 16:02:01,680 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47991.23 MB 2025-02-14 16:02:01,680 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47991.23 MB 2025-02-14 16:02:01,680 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:02:01,680 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42376.79 MB 2025-02-14 16:02:01,889 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 16:02:01,889 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 16:02:01,889 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 16:02:01,889 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:02:01,889 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40959.37 MB 2025-02-14 16:02:01,889 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43201.22 MB 2025-02-14 16:02:01,889 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 16:02:01,889 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47991.23 MB 2025-02-14 16:02:01,889 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51766.10 MB 2025-02-14 16:02:01,889 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 16:02:01,889 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48745.50 MB 2025-02-14 16:02:01,890 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 16:02:01,890 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 16:02:01,890 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 16:02:01,890 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:02:01,890 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39070.01 MB 2025-02-14 16:02:01,890 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43201.22 MB 2025-02-14 16:02:01,890 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.21 MB 2025-02-14 16:02:01,890 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47991.23 MB 2025-02-14 16:02:01,890 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51766.10 MB 2025-02-14 16:02:01,890 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3774.87 MB 2025-02-14 16:02:01,890 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48745.50 MB 2025-02-14 16:02:02,048 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 16:02:02,048 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 16:02:02,048 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 16:02:02,048 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:02:02,048 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43909.01 MB 2025-02-14 16:02:02,048 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44676.01 MB 2025-02-14 16:02:02,048 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 16:02:02,048 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51766.10 MB 2025-02-14 16:02:02,048 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52181.34 MB 2025-02-14 16:02:02,048 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 16:02:02,048 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45383.80 MB 2025-02-14 16:02:02,066 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 16:02:02,066 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 16:02:02,066 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 16:02:02,066 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:02:02,066 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45088.90 MB 2025-02-14 16:02:02,066 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45294.85 MB 2025-02-14 16:02:02,066 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.95 MB 2025-02-14 16:02:02,066 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52181.34 MB 2025-02-14 16:02:02,066 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52181.34 MB 2025-02-14 16:02:02,066 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:02:02,066 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45516.35 MB 2025-02-14 16:02:02,068 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 16:02:02,068 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 16:02:02,068 - resource_logging.py:150 - __exit__ - DEBUG - Time: 29.11 seconds 2025-02-14 16:02:02,068 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:02:02,068 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32760.32 MB 2025-02-14 16:02:02,068 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45494.94 MB 2025-02-14 16:02:02,068 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12734.61 MB 2025-02-14 16:02:02,068 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57623.45 MB 2025-02-14 16:02:02,068 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52181.34 MB 2025-02-14 16:02:02,068 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5442.11 MB 2025-02-14 16:02:02,068 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45516.35 MB 2025-02-14 16:02:02,334 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 16:02:02,334 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 16:02:02,334 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 16:02:02,334 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:02:02,334 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45494.94 MB 2025-02-14 16:02:02,334 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45594.91 MB 2025-02-14 16:02:02,334 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 99.98 MB 2025-02-14 16:02:02,334 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52181.34 MB 2025-02-14 16:02:02,334 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52181.34 MB 2025-02-14 16:02:02,334 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:02:02,334 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46194.76 MB 2025-02-14 16:02:02,351 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8122, cut from 8124 2025-02-14 16:02:02,352 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 16:02:02,358 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 16:02:02,358 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 16:02:02,358 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 16:02:02,358 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:02:02,358 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34022.26 MB 2025-02-14 16:02:02,358 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38196.23 MB 2025-02-14 16:02:02,358 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4173.96 MB 2025-02-14 16:02:02,358 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52181.34 MB 2025-02-14 16:02:02,358 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 60532.20 MB 2025-02-14 16:02:02,358 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8350.86 MB 2025-02-14 16:02:02,358 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42369.68 MB 2025-02-14 16:02:02,509 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7914] 2025-02-14 16:02:02,510 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:02:02,510 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:02:02,511 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:02:02,511 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 16:02:02,515 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 16:02:02,516 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:02:02,516 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 16:02:02,517 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 16:02:02,517 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:02:02,517 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:02:02,518 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:02:02,518 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:02:02,523 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 16:02:02,524 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:02:02,524 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:02:02,524 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:02:02,525 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:02:02,525 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 16:02:02,525 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:02:02,525 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:02:02,525 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:02:02,525 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 16:02:02,526 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 16:02:02,526 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:02:02,526 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:02:02,530 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:02:02,530 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:02:02,531 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:02:02,531 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:02:02,532 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:02:02,532 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:02:02,537 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:02:02,537 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:02:57,255 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:02:57,255 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:02:57,260 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 16:02:57,261 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:02:57,261 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1175, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 16:02:57,262 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:02:57,262 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1175, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 16:03:15,356 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 16:03:15,357 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 16:03:15,357 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.09 seconds 2025-02-14 16:03:15,357 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:03:15,357 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29546.21 MB 2025-02-14 16:03:15,357 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33704.79 MB 2025-02-14 16:03:15,357 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4158.59 MB 2025-02-14 16:03:15,357 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 66368.57 MB 2025-02-14 16:03:15,357 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38979.76 MB 2025-02-14 16:03:15,357 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -27388.81 MB 2025-02-14 16:03:15,357 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42642.20 MB 2025-02-14 16:03:15,432 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 16:03:15,432 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 16:03:15,432 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 16:03:15,432 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:03:15,432 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33704.79 MB 2025-02-14 16:03:15,432 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30276.22 MB 2025-02-14 16:03:15,432 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3428.57 MB 2025-02-14 16:03:15,432 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38979.76 MB 2025-02-14 16:03:15,432 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53351.55 MB 2025-02-14 16:03:15,432 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 14371.78 MB 2025-02-14 16:03:15,432 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46207.56 MB 2025-02-14 16:03:17,349 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 16:03:17,349 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 16:03:17,349 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 16:03:17,349 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:03:17,349 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30276.22 MB 2025-02-14 16:03:17,349 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30807.06 MB 2025-02-14 16:03:17,349 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 16:03:17,349 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53351.55 MB 2025-02-14 16:03:17,349 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34141.63 MB 2025-02-14 16:03:17,349 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -19209.91 MB 2025-02-14 16:03:17,349 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34787.44 MB 2025-02-14 16:03:17,377 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 16:03:17,377 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 16:03:17,377 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 16:03:17,377 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:03:17,377 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30807.06 MB 2025-02-14 16:03:17,377 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32696.60 MB 2025-02-14 16:03:17,377 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 16:03:17,377 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34141.63 MB 2025-02-14 16:03:17,377 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36972.79 MB 2025-02-14 16:03:17,377 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-14 16:03:17,377 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34114.03 MB 2025-02-14 16:03:17,593 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 16:03:17,593 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 16:03:17,593 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 16:03:17,593 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:03:17,593 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32696.60 MB 2025-02-14 16:03:17,593 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34938.45 MB 2025-02-14 16:03:17,593 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 16:03:17,593 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36972.79 MB 2025-02-14 16:03:17,593 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42635.10 MB 2025-02-14 16:03:17,593 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 16:03:17,593 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40482.74 MB 2025-02-14 16:03:17,594 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 16:03:17,594 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 16:03:17,594 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.24 seconds 2025-02-14 16:03:17,594 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:03:17,594 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30807.06 MB 2025-02-14 16:03:17,594 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34938.45 MB 2025-02-14 16:03:17,594 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 16:03:17,594 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34141.63 MB 2025-02-14 16:03:17,594 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42635.10 MB 2025-02-14 16:03:17,594 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8493.47 MB 2025-02-14 16:03:17,594 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40482.74 MB 2025-02-14 16:03:17,757 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 16:03:17,757 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 16:03:17,757 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 16:03:17,757 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:03:17,757 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35646.24 MB 2025-02-14 16:03:17,757 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36413.24 MB 2025-02-14 16:03:17,757 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 16:03:17,757 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42635.10 MB 2025-02-14 16:03:17,757 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43052.43 MB 2025-02-14 16:03:17,757 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 16:03:17,757 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37121.03 MB 2025-02-14 16:03:17,775 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 16:03:17,775 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 16:03:17,775 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 16:03:17,775 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:03:17,775 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36826.13 MB 2025-02-14 16:03:17,775 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37032.77 MB 2025-02-14 16:03:17,775 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.63 MB 2025-02-14 16:03:17,775 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43052.43 MB 2025-02-14 16:03:17,775 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43052.43 MB 2025-02-14 16:03:17,775 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:03:17,775 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37252.51 MB 2025-02-14 16:03:17,777 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 16:03:17,777 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 16:03:17,777 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.51 seconds 2025-02-14 16:03:17,777 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:03:17,777 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25452.41 MB 2025-02-14 16:03:17,777 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37233.67 MB 2025-02-14 16:03:17,777 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11781.25 MB 2025-02-14 16:03:17,777 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 66368.57 MB 2025-02-14 16:03:17,777 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43052.43 MB 2025-02-14 16:03:17,777 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23316.14 MB 2025-02-14 16:03:17,777 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37252.51 MB 2025-02-14 16:03:18,040 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 16:03:18,040 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 16:03:18,040 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 16:03:18,040 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:03:18,040 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26614.78 MB 2025-02-14 16:03:18,040 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26715.40 MB 2025-02-14 16:03:18,040 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.62 MB 2025-02-14 16:03:18,040 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43052.43 MB 2025-02-14 16:03:18,040 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43052.43 MB 2025-02-14 16:03:18,040 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:03:18,040 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27318.03 MB 2025-02-14 16:03:18,058 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8155, cut from 8157 2025-02-14 16:03:18,058 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 16:03:18,064 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 16:03:18,064 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 16:03:18,064 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 16:03:18,064 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:03:18,064 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26715.40 MB 2025-02-14 16:03:18,064 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30906.30 MB 2025-02-14 16:03:18,065 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4190.89 MB 2025-02-14 16:03:18,065 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43052.43 MB 2025-02-14 16:03:18,065 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51436.85 MB 2025-02-14 16:03:18,065 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8384.41 MB 2025-02-14 16:03:18,065 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35096.68 MB 2025-02-14 16:03:18,228 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7947] 2025-02-14 16:03:18,229 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:03:18,229 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:03:18,230 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:03:18,230 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 16:03:18,235 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 16:03:18,236 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:03:18,236 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 16:03:18,236 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 16:03:18,237 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:03:18,237 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:03:18,237 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:03:18,237 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:03:18,243 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 16:03:18,244 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:03:18,244 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:03:18,244 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:03:18,244 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:03:18,244 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 16:03:18,245 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:03:18,245 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:03:18,245 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:03:18,245 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 16:03:18,245 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 16:03:18,246 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:03:18,246 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:03:18,252 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:03:18,252 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:03:18,254 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:03:18,254 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:03:18,256 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:03:18,256 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:03:18,263 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:03:18,263 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:03:49,589 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:03:49,589 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:03:49,594 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 16:03:49,595 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:03:49,595 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1238, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 16:03:49,596 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:03:49,596 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1238, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 16:04:08,750 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 16:04:08,751 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 16:04:08,751 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.15 seconds 2025-02-14 16:04:08,751 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:04:08,751 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30107.13 MB 2025-02-14 16:04:08,751 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34488.34 MB 2025-02-14 16:04:08,751 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4381.21 MB 2025-02-14 16:04:08,751 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57394.86 MB 2025-02-14 16:04:08,751 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41422.95 MB 2025-02-14 16:04:08,751 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15971.91 MB 2025-02-14 16:04:08,751 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43428.87 MB 2025-02-14 16:04:08,828 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 16:04:08,828 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 16:04:08,828 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 16:04:08,828 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:04:08,828 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34488.34 MB 2025-02-14 16:04:08,828 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30725.66 MB 2025-02-14 16:04:08,828 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3762.67 MB 2025-02-14 16:04:08,828 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41422.95 MB 2025-02-14 16:04:08,828 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53445.92 MB 2025-02-14 16:04:08,828 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 12022.97 MB 2025-02-14 16:04:08,828 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47417.22 MB 2025-02-14 16:04:10,760 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 16:04:10,760 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 16:04:10,760 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 16:04:10,760 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:04:10,760 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30725.66 MB 2025-02-14 16:04:10,760 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31256.51 MB 2025-02-14 16:04:10,760 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 16:04:10,760 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53445.92 MB 2025-02-14 16:04:10,760 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35483.81 MB 2025-02-14 16:04:10,760 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17962.11 MB 2025-02-14 16:04:10,760 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35235.84 MB 2025-02-14 16:04:10,774 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 16:04:10,774 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 16:04:10,774 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 16:04:10,774 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:04:10,774 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31256.51 MB 2025-02-14 16:04:10,774 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33146.04 MB 2025-02-14 16:04:10,774 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 16:04:10,774 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35483.81 MB 2025-02-14 16:04:10,774 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37371.25 MB 2025-02-14 16:04:10,774 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 16:04:10,774 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34563.47 MB 2025-02-14 16:04:10,984 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 16:04:10,984 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 16:04:10,984 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 16:04:10,984 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:04:10,984 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33146.04 MB 2025-02-14 16:04:10,985 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35387.90 MB 2025-02-14 16:04:10,985 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 16:04:10,985 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37371.25 MB 2025-02-14 16:04:10,985 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43033.56 MB 2025-02-14 16:04:10,985 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 16:04:10,985 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40932.18 MB 2025-02-14 16:04:10,985 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 16:04:10,985 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 16:04:10,985 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 16:04:10,985 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:04:10,985 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31256.51 MB 2025-02-14 16:04:10,985 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35387.90 MB 2025-02-14 16:04:10,985 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 16:04:10,985 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35483.81 MB 2025-02-14 16:04:10,985 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43033.56 MB 2025-02-14 16:04:10,985 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-14 16:04:10,985 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40932.18 MB 2025-02-14 16:04:11,152 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 16:04:11,152 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 16:04:11,152 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 16:04:11,152 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:04:11,152 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36095.69 MB 2025-02-14 16:04:11,152 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36862.69 MB 2025-02-14 16:04:11,152 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 16:04:11,152 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43033.56 MB 2025-02-14 16:04:11,152 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43450.89 MB 2025-02-14 16:04:11,152 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 16:04:11,152 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37570.48 MB 2025-02-14 16:04:11,170 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 16:04:11,170 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 16:04:11,170 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 16:04:11,170 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:04:11,170 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37275.58 MB 2025-02-14 16:04:11,170 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37481.29 MB 2025-02-14 16:04:11,170 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.71 MB 2025-02-14 16:04:11,170 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43450.89 MB 2025-02-14 16:04:11,170 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43450.89 MB 2025-02-14 16:04:11,170 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:04:11,170 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37697.60 MB 2025-02-14 16:04:11,171 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 16:04:11,171 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 16:04:11,171 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.57 seconds 2025-02-14 16:04:11,171 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:04:11,171 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25793.84 MB 2025-02-14 16:04:11,171 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37681.57 MB 2025-02-14 16:04:11,172 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11887.74 MB 2025-02-14 16:04:11,172 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57394.86 MB 2025-02-14 16:04:11,172 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43450.89 MB 2025-02-14 16:04:11,172 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13943.96 MB 2025-02-14 16:04:11,172 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37697.60 MB 2025-02-14 16:04:11,437 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 16:04:11,437 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 16:04:11,437 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 16:04:11,437 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:04:11,437 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37681.57 MB 2025-02-14 16:04:11,437 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37781.65 MB 2025-02-14 16:04:11,437 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.07 MB 2025-02-14 16:04:11,437 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43450.89 MB 2025-02-14 16:04:11,437 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43450.89 MB 2025-02-14 16:04:11,437 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:04:11,437 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38382.09 MB 2025-02-14 16:04:11,455 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8130, cut from 8132 2025-02-14 16:04:11,456 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 16:04:11,462 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 16:04:11,462 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 16:04:11,462 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 16:04:11,462 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:04:11,462 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27055.97 MB 2025-02-14 16:04:11,462 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31234.04 MB 2025-02-14 16:04:11,462 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4178.07 MB 2025-02-14 16:04:11,462 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43450.89 MB 2025-02-14 16:04:11,462 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51810.14 MB 2025-02-14 16:04:11,462 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8359.25 MB 2025-02-14 16:04:11,462 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35411.59 MB 2025-02-14 16:04:11,624 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7922] 2025-02-14 16:04:11,625 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:04:11,625 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:04:11,626 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:04:11,626 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 16:04:11,631 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 16:04:11,632 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:04:11,632 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 16:04:11,632 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 16:04:11,632 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:04:11,633 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:04:11,633 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:04:11,633 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:04:11,639 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 16:04:11,639 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:04:11,639 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:04:11,640 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:04:11,640 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:04:11,640 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 16:04:11,640 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:04:11,640 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:04:11,641 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:04:11,641 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 16:04:11,641 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 16:04:11,641 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:04:11,641 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:04:11,647 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:04:11,647 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:04:11,648 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:04:11,648 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:04:11,649 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:04:11,649 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:04:11,655 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:04:11,655 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:05:02,205 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:05:02,206 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:05:02,210 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 16:05:02,212 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:05:02,212 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 626, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 16:05:02,213 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:05:02,213 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 626, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 16:05:11,874 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 16:05:11,874 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 16:05:11,874 - resource_logging.py:150 - __exit__ - DEBUG - Time: 9.66 seconds 2025-02-14 16:05:11,874 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:05:11,874 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25964.24 MB 2025-02-14 16:05:11,874 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28179.61 MB 2025-02-14 16:05:11,874 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2215.38 MB 2025-02-14 16:05:11,874 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57889.78 MB 2025-02-14 16:05:11,874 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 34191.97 MB 2025-02-14 16:05:11,874 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -23697.82 MB 2025-02-14 16:05:11,874 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37021.05 MB 2025-02-14 16:05:11,916 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 16:05:11,916 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 16:05:11,916 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-14 16:05:11,916 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:05:11,916 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28179.61 MB 2025-02-14 16:05:11,916 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27665.69 MB 2025-02-14 16:05:11,916 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -513.92 MB 2025-02-14 16:05:11,916 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 34191.97 MB 2025-02-14 16:05:11,916 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39984.30 MB 2025-02-14 16:05:11,916 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5792.33 MB 2025-02-14 16:05:11,916 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36365.44 MB 2025-02-14 16:05:13,821 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 16:05:13,821 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 16:05:13,821 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.90 seconds 2025-02-14 16:05:13,821 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:05:13,822 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27665.69 MB 2025-02-14 16:05:13,822 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28196.53 MB 2025-02-14 16:05:13,822 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 16:05:13,822 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39984.30 MB 2025-02-14 16:05:13,822 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35607.54 MB 2025-02-14 16:05:13,822 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4376.76 MB 2025-02-14 16:05:13,822 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32175.87 MB 2025-02-14 16:05:13,835 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 16:05:13,835 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 16:05:13,835 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 16:05:13,835 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:05:13,835 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28196.53 MB 2025-02-14 16:05:13,835 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30086.07 MB 2025-02-14 16:05:13,835 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 16:05:13,835 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35607.54 MB 2025-02-14 16:05:13,835 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35607.54 MB 2025-02-14 16:05:13,835 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:05:13,835 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31503.50 MB 2025-02-14 16:05:14,043 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 16:05:14,043 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 16:05:14,043 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 16:05:14,043 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:05:14,043 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30086.07 MB 2025-02-14 16:05:14,043 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32327.92 MB 2025-02-14 16:05:14,043 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 16:05:14,043 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35607.54 MB 2025-02-14 16:05:14,043 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40798.00 MB 2025-02-14 16:05:14,043 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 16:05:14,043 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37872.20 MB 2025-02-14 16:05:14,043 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 16:05:14,043 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 16:05:14,043 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 16:05:14,043 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:05:14,043 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28196.53 MB 2025-02-14 16:05:14,043 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32327.92 MB 2025-02-14 16:05:14,043 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 16:05:14,043 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35607.54 MB 2025-02-14 16:05:14,043 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40798.00 MB 2025-02-14 16:05:14,044 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 16:05:14,044 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37872.20 MB 2025-02-14 16:05:14,204 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 16:05:14,204 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 16:05:14,204 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 16:05:14,204 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:05:14,204 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33035.71 MB 2025-02-14 16:05:14,204 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33802.71 MB 2025-02-14 16:05:14,204 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 16:05:14,204 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40798.00 MB 2025-02-14 16:05:14,204 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41213.23 MB 2025-02-14 16:05:14,204 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 16:05:14,204 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34510.50 MB 2025-02-14 16:05:14,221 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 16:05:14,221 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 16:05:14,221 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 16:05:14,221 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:05:14,221 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34215.60 MB 2025-02-14 16:05:14,221 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34425.02 MB 2025-02-14 16:05:14,221 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 209.42 MB 2025-02-14 16:05:14,221 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41213.23 MB 2025-02-14 16:05:14,221 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41213.23 MB 2025-02-14 16:05:14,221 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:05:14,221 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34603.81 MB 2025-02-14 16:05:14,222 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 16:05:14,222 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 16:05:14,222 - resource_logging.py:150 - __exit__ - DEBUG - Time: 12.01 seconds 2025-02-14 16:05:14,222 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:05:14,222 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23783.20 MB 2025-02-14 16:05:14,222 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34626.09 MB 2025-02-14 16:05:14,222 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 10842.89 MB 2025-02-14 16:05:14,222 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57889.78 MB 2025-02-14 16:05:14,222 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41213.23 MB 2025-02-14 16:05:14,222 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16676.55 MB 2025-02-14 16:05:14,222 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34626.09 MB 2025-02-14 16:05:14,484 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 16:05:14,484 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 16:05:14,484 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 16:05:14,484 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:05:14,484 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34626.09 MB 2025-02-14 16:05:14,484 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34726.56 MB 2025-02-14 16:05:14,484 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.47 MB 2025-02-14 16:05:14,484 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41213.23 MB 2025-02-14 16:05:14,484 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41213.23 MB 2025-02-14 16:05:14,484 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:05:14,484 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35329.36 MB 2025-02-14 16:05:14,502 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 16:05:14,502 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 16:05:14,508 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 16:05:14,508 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 16:05:14,509 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 16:05:14,509 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:05:14,509 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25046.12 MB 2025-02-14 16:05:14,509 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29240.61 MB 2025-02-14 16:05:14,509 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4194.49 MB 2025-02-14 16:05:14,509 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41213.23 MB 2025-02-14 16:05:14,509 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49603.94 MB 2025-02-14 16:05:14,509 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 16:05:14,509 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33434.91 MB 2025-02-14 16:05:14,664 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 16:05:14,665 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:05:14,665 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:05:14,666 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:05:14,666 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 16:05:14,671 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 16:05:14,672 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:05:14,672 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 16:05:14,672 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 16:05:14,672 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:05:14,673 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:05:14,673 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:05:14,673 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:05:14,679 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 16:05:14,679 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:05:14,679 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:05:14,680 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:05:14,680 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:05:14,680 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 16:05:14,680 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:05:14,680 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:05:14,681 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:05:14,681 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 16:05:14,681 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 16:05:14,681 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:05:14,681 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:05:14,685 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:05:14,685 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:05:14,686 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:05:14,686 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:05:14,688 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:05:14,688 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:05:14,694 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:05:14,694 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:05:18,320 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:05:18,321 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:05:18,325 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 16:05:18,326 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:05:18,326 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1301, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 16:05:18,327 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:05:18,327 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1301, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 16:05:38,527 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 16:05:38,528 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 16:05:38,528 - resource_logging.py:150 - __exit__ - DEBUG - Time: 20.19 seconds 2025-02-14 16:05:38,528 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:05:38,528 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30789.52 MB 2025-02-14 16:05:38,528 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35393.68 MB 2025-02-14 16:05:38,528 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4604.17 MB 2025-02-14 16:05:38,528 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55805.21 MB 2025-02-14 16:05:38,528 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47309.65 MB 2025-02-14 16:05:38,528 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8495.56 MB 2025-02-14 16:05:38,528 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44337.75 MB 2025-02-14 16:05:38,607 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 16:05:38,607 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 16:05:38,607 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 16:05:38,607 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:05:38,607 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35393.68 MB 2025-02-14 16:05:38,607 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31296.58 MB 2025-02-14 16:05:38,607 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4097.10 MB 2025-02-14 16:05:38,607 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47309.65 MB 2025-02-14 16:05:38,607 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56287.56 MB 2025-02-14 16:05:38,607 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8977.91 MB 2025-02-14 16:05:38,607 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48910.66 MB 2025-02-14 16:05:40,586 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 16:05:40,586 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 16:05:40,586 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.98 seconds 2025-02-14 16:05:40,586 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:05:40,586 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31296.58 MB 2025-02-14 16:05:40,586 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31827.42 MB 2025-02-14 16:05:40,586 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 16:05:40,586 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56287.56 MB 2025-02-14 16:05:40,586 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42704.31 MB 2025-02-14 16:05:40,586 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13583.25 MB 2025-02-14 16:05:40,586 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35806.75 MB 2025-02-14 16:05:40,599 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 16:05:40,600 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 16:05:40,600 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 16:05:40,600 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:05:40,600 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31827.42 MB 2025-02-14 16:05:40,600 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33716.95 MB 2025-02-14 16:05:40,600 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 16:05:40,600 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42704.31 MB 2025-02-14 16:05:40,600 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42704.31 MB 2025-02-14 16:05:40,600 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:05:40,600 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35134.38 MB 2025-02-14 16:05:40,813 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 16:05:40,813 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 16:05:40,813 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 16:05:40,813 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:05:40,813 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33716.95 MB 2025-02-14 16:05:40,813 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35958.81 MB 2025-02-14 16:05:40,813 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 16:05:40,813 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42704.31 MB 2025-02-14 16:05:40,813 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45063.60 MB 2025-02-14 16:05:40,813 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2359.30 MB 2025-02-14 16:05:40,813 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41503.09 MB 2025-02-14 16:05:40,814 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 16:05:40,814 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 16:05:40,814 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 16:05:40,814 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:05:40,814 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31827.42 MB 2025-02-14 16:05:40,814 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35958.81 MB 2025-02-14 16:05:40,814 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 16:05:40,814 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42704.31 MB 2025-02-14 16:05:40,814 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45063.60 MB 2025-02-14 16:05:40,814 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2359.30 MB 2025-02-14 16:05:40,814 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41503.09 MB 2025-02-14 16:05:40,979 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 16:05:40,979 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 16:05:40,979 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 16:05:40,979 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:05:40,979 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36666.60 MB 2025-02-14 16:05:40,979 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37433.60 MB 2025-02-14 16:05:40,980 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 16:05:40,980 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45063.60 MB 2025-02-14 16:05:40,980 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45478.84 MB 2025-02-14 16:05:40,980 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 16:05:40,980 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38141.39 MB 2025-02-14 16:05:40,997 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 16:05:40,997 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 16:05:40,997 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 16:05:40,997 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:05:40,997 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37846.49 MB 2025-02-14 16:05:40,997 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38051.87 MB 2025-02-14 16:05:40,997 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.38 MB 2025-02-14 16:05:40,997 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45478.84 MB 2025-02-14 16:05:40,997 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45478.84 MB 2025-02-14 16:05:40,997 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:05:40,997 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38271.18 MB 2025-02-14 16:05:40,998 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 16:05:40,998 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 16:05:40,998 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.67 seconds 2025-02-14 16:05:40,998 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:05:40,998 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26256.73 MB 2025-02-14 16:05:40,998 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38252.18 MB 2025-02-14 16:05:40,998 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11995.45 MB 2025-02-14 16:05:40,998 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55805.21 MB 2025-02-14 16:05:40,998 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45478.84 MB 2025-02-14 16:05:40,998 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10326.38 MB 2025-02-14 16:05:40,998 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38271.18 MB 2025-02-14 16:05:41,265 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 16:05:41,265 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 16:05:41,265 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 16:05:41,265 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:05:41,265 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38252.18 MB 2025-02-14 16:05:41,265 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38352.27 MB 2025-02-14 16:05:41,265 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.09 MB 2025-02-14 16:05:41,265 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45478.84 MB 2025-02-14 16:05:41,265 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45478.84 MB 2025-02-14 16:05:41,265 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:05:41,265 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38952.78 MB 2025-02-14 16:05:41,283 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8131, cut from 8133 2025-02-14 16:05:41,283 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 16:05:41,290 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 16:05:41,290 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 16:05:41,290 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 16:05:41,290 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:05:41,290 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27518.89 MB 2025-02-14 16:05:41,290 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31697.47 MB 2025-02-14 16:05:41,290 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4178.58 MB 2025-02-14 16:05:41,290 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45478.84 MB 2025-02-14 16:05:41,290 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49658.46 MB 2025-02-14 16:05:41,290 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4179.62 MB 2025-02-14 16:05:41,290 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35875.54 MB 2025-02-14 16:05:41,457 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7923] 2025-02-14 16:05:41,458 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:05:41,459 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:05:41,459 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:05:41,459 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 16:05:41,464 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 16:05:41,465 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:05:41,465 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 16:05:41,465 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 16:05:41,466 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:05:41,466 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:05:41,467 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:05:41,467 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:05:41,473 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 16:05:41,473 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:05:41,473 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:05:41,474 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:05:41,474 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:05:41,474 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 16:05:41,474 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:05:41,474 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:05:41,475 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:05:41,475 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 16:05:41,475 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 16:05:41,475 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:05:41,475 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:05:41,481 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:05:41,481 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:05:41,483 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:05:41,483 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:05:41,485 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:05:41,485 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:05:41,492 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:05:41,492 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:05:54,013 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:05:54,013 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:05:54,017 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 16:05:54,018 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:05:54,018 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 102, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 16:05:54,019 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:05:54,019 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 102, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 16:05:55,627 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 16:05:55,627 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 16:05:55,627 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.60 seconds 2025-02-14 16:05:55,627 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:05:55,627 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22556.47 MB 2025-02-14 16:05:55,627 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22917.44 MB 2025-02-14 16:05:55,627 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 360.97 MB 2025-02-14 16:05:55,627 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55981.38 MB 2025-02-14 16:05:55,627 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25119.69 MB 2025-02-14 16:05:55,627 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -30861.69 MB 2025-02-14 16:05:55,627 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31801.35 MB 2025-02-14 16:05:55,630 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 16:05:55,630 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 16:05:55,630 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 16:05:55,630 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:05:55,630 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22917.44 MB 2025-02-14 16:05:55,630 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23092.33 MB 2025-02-14 16:05:55,630 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 174.89 MB 2025-02-14 16:05:55,630 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25119.69 MB 2025-02-14 16:05:55,630 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25119.69 MB 2025-02-14 16:05:55,630 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:05:55,630 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23633.85 MB 2025-02-14 16:05:56,136 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 16:05:56,136 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 16:05:56,136 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.50 seconds 2025-02-14 16:05:56,137 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:05:56,137 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23092.33 MB 2025-02-14 16:05:56,137 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23227.70 MB 2025-02-14 16:05:56,137 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 135.36 MB 2025-02-14 16:05:56,137 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25119.69 MB 2025-02-14 16:05:56,137 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25119.69 MB 2025-02-14 16:05:56,137 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:05:56,137 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27179.91 MB 2025-02-14 16:05:56,143 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 16:05:56,143 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 16:05:56,143 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 16:05:56,143 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:05:56,143 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23227.63 MB 2025-02-14 16:05:56,143 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23709.34 MB 2025-02-14 16:05:56,143 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 481.71 MB 2025-02-14 16:05:56,143 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25119.69 MB 2025-02-14 16:05:56,143 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25119.69 MB 2025-02-14 16:05:56,143 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:05:56,143 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24070.79 MB 2025-02-14 16:05:56,245 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 16:05:56,245 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 16:05:56,245 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 16:05:56,245 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:05:56,245 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23709.34 MB 2025-02-14 16:05:56,245 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24294.43 MB 2025-02-14 16:05:56,245 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 585.09 MB 2025-02-14 16:05:56,245 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25119.69 MB 2025-02-14 16:05:56,245 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26447.18 MB 2025-02-14 16:05:56,245 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1327.50 MB 2025-02-14 16:05:56,245 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25696.91 MB 2025-02-14 16:05:56,246 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 16:05:56,246 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 16:05:56,246 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 16:05:56,246 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:05:56,246 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23227.63 MB 2025-02-14 16:05:56,246 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24294.43 MB 2025-02-14 16:05:56,246 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1066.80 MB 2025-02-14 16:05:56,246 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25119.69 MB 2025-02-14 16:05:56,246 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26447.18 MB 2025-02-14 16:05:56,246 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1327.50 MB 2025-02-14 16:05:56,246 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25696.91 MB 2025-02-14 16:05:56,293 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 16:05:56,293 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 16:05:56,293 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.04 seconds 2025-02-14 16:05:56,293 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:05:56,293 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24555.14 MB 2025-02-14 16:05:56,293 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24800.86 MB 2025-02-14 16:05:56,293 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 245.72 MB 2025-02-14 16:05:56,293 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26447.18 MB 2025-02-14 16:05:56,293 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26604.47 MB 2025-02-14 16:05:56,293 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 157.29 MB 2025-02-14 16:05:56,293 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24981.34 MB 2025-02-14 16:05:56,299 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 16:05:56,299 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 16:05:56,299 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 16:05:56,299 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:05:56,299 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24956.29 MB 2025-02-14 16:05:56,299 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25155.34 MB 2025-02-14 16:05:56,299 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 199.05 MB 2025-02-14 16:05:56,299 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26604.47 MB 2025-02-14 16:05:56,299 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26604.47 MB 2025-02-14 16:05:56,299 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:05:56,299 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25155.34 MB 2025-02-14 16:05:56,301 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 16:05:56,301 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 16:05:56,301 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.28 seconds 2025-02-14 16:05:56,301 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:05:56,301 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22201.09 MB 2025-02-14 16:05:56,301 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25355.65 MB 2025-02-14 16:05:56,301 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3154.56 MB 2025-02-14 16:05:56,301 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55981.38 MB 2025-02-14 16:05:56,301 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26604.47 MB 2025-02-14 16:05:56,301 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29376.91 MB 2025-02-14 16:05:56,301 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25355.65 MB 2025-02-14 16:05:56,565 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 16:05:56,565 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 16:05:56,565 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 16:05:56,565 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:05:56,565 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22572.04 MB 2025-02-14 16:05:56,565 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 22672.13 MB 2025-02-14 16:05:56,565 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.09 MB 2025-02-14 16:05:56,565 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26604.47 MB 2025-02-14 16:05:56,565 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26604.47 MB 2025-02-14 16:05:56,565 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:05:56,565 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23272.64 MB 2025-02-14 16:05:56,583 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8131, cut from 8133 2025-02-14 16:05:56,583 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 16:05:56,589 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 16:05:56,589 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 16:05:56,589 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 16:05:56,589 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:05:56,589 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22672.13 MB 2025-02-14 16:05:56,589 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26851.75 MB 2025-02-14 16:05:56,589 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4179.62 MB 2025-02-14 16:05:56,589 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26604.47 MB 2025-02-14 16:05:56,589 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37054.58 MB 2025-02-14 16:05:56,589 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10450.11 MB 2025-02-14 16:05:56,589 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31029.82 MB 2025-02-14 16:05:56,739 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7923] 2025-02-14 16:05:56,741 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:05:56,741 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:05:56,741 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:05:56,741 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 16:05:56,746 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 16:05:56,747 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:05:56,747 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 16:05:56,747 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 16:05:56,748 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:05:56,748 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:05:56,748 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:05:56,748 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:05:56,754 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 16:05:56,754 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:05:56,754 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:05:56,755 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:05:56,755 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:05:56,755 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 16:05:56,755 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:05:56,755 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:05:56,756 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:05:56,756 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 16:05:56,756 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 16:05:56,756 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:05:56,756 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:05:56,759 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:05:56,759 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:05:56,760 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:05:56,760 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:05:56,761 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:05:56,761 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:05:56,766 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:05:56,766 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:06:45,367 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:06:45,367 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:06:45,371 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 16:06:45,373 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:06:45,373 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 239, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 16:06:45,374 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:06:45,374 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 239, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 16:06:49,044 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 16:06:49,044 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 16:06:49,044 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.67 seconds 2025-02-14 16:06:49,044 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:06:49,044 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28760.80 MB 2025-02-14 16:06:49,044 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29606.60 MB 2025-02-14 16:06:49,044 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 845.81 MB 2025-02-14 16:06:49,044 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43499.13 MB 2025-02-14 16:06:49,044 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32728.15 MB 2025-02-14 16:06:49,044 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10770.97 MB 2025-02-14 16:06:49,044 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38459.47 MB 2025-02-14 16:06:49,060 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 16:06:49,061 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 16:06:49,061 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 16:06:49,061 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:06:49,061 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29606.60 MB 2025-02-14 16:06:49,061 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24888.41 MB 2025-02-14 16:06:49,061 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4718.19 MB 2025-02-14 16:06:49,061 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32728.15 MB 2025-02-14 16:06:49,061 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 32728.15 MB 2025-02-14 16:06:49,061 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:06:49,061 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31124.88 MB 2025-02-14 16:06:50,194 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 16:06:50,195 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 16:06:50,195 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.13 seconds 2025-02-14 16:06:50,195 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:06:50,195 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24888.41 MB 2025-02-14 16:06:50,195 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25205.59 MB 2025-02-14 16:06:50,195 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 317.18 MB 2025-02-14 16:06:50,195 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 32728.15 MB 2025-02-14 16:06:50,195 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31518.10 MB 2025-02-14 16:06:50,195 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1210.06 MB 2025-02-14 16:06:50,195 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29143.78 MB 2025-02-14 16:06:50,204 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 16:06:50,204 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 16:06:50,204 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 16:06:50,204 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:06:50,204 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25205.59 MB 2025-02-14 16:06:50,204 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26334.31 MB 2025-02-14 16:06:50,204 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1128.72 MB 2025-02-14 16:06:50,204 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31518.10 MB 2025-02-14 16:06:50,204 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 31518.10 MB 2025-02-14 16:06:50,204 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:06:50,204 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27181.23 MB 2025-02-14 16:06:50,329 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 16:06:50,329 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 16:06:50,329 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 16:06:50,329 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:06:50,329 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26334.31 MB 2025-02-14 16:06:50,329 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27673.85 MB 2025-02-14 16:06:50,329 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1339.53 MB 2025-02-14 16:06:50,329 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31518.10 MB 2025-02-14 16:06:50,329 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33210.50 MB 2025-02-14 16:06:50,329 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1692.40 MB 2025-02-14 16:06:50,329 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30987.32 MB 2025-02-14 16:06:50,330 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 16:06:50,330 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 16:06:50,330 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 16:06:50,330 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:06:50,330 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25205.59 MB 2025-02-14 16:06:50,330 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27673.85 MB 2025-02-14 16:06:50,330 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2468.26 MB 2025-02-14 16:06:50,330 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 31518.10 MB 2025-02-14 16:06:50,330 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33210.50 MB 2025-02-14 16:06:50,330 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1692.40 MB 2025-02-14 16:06:50,330 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30987.32 MB 2025-02-14 16:06:50,424 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 16:06:50,424 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 16:06:50,424 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 16:06:50,424 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:06:50,424 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28096.75 MB 2025-02-14 16:06:50,424 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28555.04 MB 2025-02-14 16:06:50,424 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 458.28 MB 2025-02-14 16:06:50,424 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33210.50 MB 2025-02-14 16:06:50,424 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33451.67 MB 2025-02-14 16:06:50,424 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 241.17 MB 2025-02-14 16:06:50,424 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28977.94 MB 2025-02-14 16:06:50,435 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 16:06:50,435 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 16:06:50,435 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 16:06:50,435 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:06:50,435 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28801.74 MB 2025-02-14 16:06:50,435 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29007.82 MB 2025-02-14 16:06:50,435 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.08 MB 2025-02-14 16:06:50,435 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33451.67 MB 2025-02-14 16:06:50,435 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33451.67 MB 2025-02-14 16:06:50,435 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:06:50,435 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29091.56 MB 2025-02-14 16:06:50,436 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 16:06:50,436 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 16:06:50,436 - resource_logging.py:150 - __exit__ - DEBUG - Time: 5.06 seconds 2025-02-14 16:06:50,436 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:06:50,436 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27928.10 MB 2025-02-14 16:06:50,436 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29208.67 MB 2025-02-14 16:06:50,436 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1280.57 MB 2025-02-14 16:06:50,436 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43499.13 MB 2025-02-14 16:06:50,436 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33451.67 MB 2025-02-14 16:06:50,436 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10047.46 MB 2025-02-14 16:06:50,436 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29208.67 MB 2025-02-14 16:06:50,697 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 16:06:50,698 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 16:06:50,698 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 16:06:50,698 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:06:50,698 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29208.67 MB 2025-02-14 16:06:50,698 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29309.03 MB 2025-02-14 16:06:50,698 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.36 MB 2025-02-14 16:06:50,698 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33451.67 MB 2025-02-14 16:06:50,698 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 33451.67 MB 2025-02-14 16:06:50,698 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:06:50,698 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29911.61 MB 2025-02-14 16:06:50,716 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8153, cut from 8155 2025-02-14 16:06:50,716 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2 ('] 2025-02-14 16:06:50,722 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 16:06:50,722 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 16:06:50,722 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 16:06:50,722 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:06:50,722 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29309.03 MB 2025-02-14 16:06:50,722 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27825.57 MB 2025-02-14 16:06:50,722 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1483.45 MB 2025-02-14 16:06:50,722 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 33451.67 MB 2025-02-14 16:06:50,722 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41831.89 MB 2025-02-14 16:06:50,722 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8380.22 MB 2025-02-14 16:06:50,722 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32015.69 MB 2025-02-14 16:06:50,873 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7945] 2025-02-14 16:06:50,874 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:06:50,874 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:06:50,875 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:06:50,875 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 16:06:50,879 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 16:06:50,880 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:06:50,880 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 16:06:50,881 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2 ('] 2025-02-14 16:06:50,881 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:06:50,881 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:06:50,882 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:06:50,882 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:06:50,887 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 16:06:50,888 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:06:50,888 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:06:50,888 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:06:50,888 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:06:50,888 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 16:06:50,889 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:06:50,889 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:06:50,889 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:06:50,889 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 16:06:50,889 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 16:06:50,890 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:06:50,890 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:06:50,893 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:06:50,893 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:06:50,894 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:06:50,894 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:06:50,895 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:06:50,895 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:06:50,901 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:06:50,901 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:07:45,310 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:07:45,311 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:07:45,318 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 16:07:45,321 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:07:45,321 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1238, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 16:07:45,322 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:07:45,322 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1238, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 16:08:04,359 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 16:08:04,359 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 16:08:04,359 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.03 seconds 2025-02-14 16:08:04,359 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:08:04,359 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30715.85 MB 2025-02-14 16:08:04,359 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35097.06 MB 2025-02-14 16:08:04,359 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4381.21 MB 2025-02-14 16:08:04,359 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48398.07 MB 2025-02-14 16:08:04,359 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48752.49 MB 2025-02-14 16:08:04,359 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 354.42 MB 2025-02-14 16:08:04,359 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44037.59 MB 2025-02-14 16:08:04,430 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 16:08:04,430 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 16:08:04,430 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 16:08:04,430 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:08:04,430 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35097.06 MB 2025-02-14 16:08:04,430 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31334.38 MB 2025-02-14 16:08:04,430 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3762.67 MB 2025-02-14 16:08:04,430 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48752.49 MB 2025-02-14 16:08:04,430 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53116.67 MB 2025-02-14 16:08:04,430 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4364.17 MB 2025-02-14 16:08:04,430 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48141.89 MB 2025-02-14 16:08:06,332 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 16:08:06,332 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 16:08:06,332 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.90 seconds 2025-02-14 16:08:06,332 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:08:06,332 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31334.38 MB 2025-02-14 16:08:06,332 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31865.23 MB 2025-02-14 16:08:06,332 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 16:08:06,332 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53116.67 MB 2025-02-14 16:08:06,332 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40179.34 MB 2025-02-14 16:08:06,332 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12937.33 MB 2025-02-14 16:08:06,332 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35844.56 MB 2025-02-14 16:08:06,346 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 16:08:06,346 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 16:08:06,346 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 16:08:06,346 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:08:06,346 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31865.23 MB 2025-02-14 16:08:06,346 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33754.76 MB 2025-02-14 16:08:06,346 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 16:08:06,346 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40179.34 MB 2025-02-14 16:08:06,346 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40179.34 MB 2025-02-14 16:08:06,346 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:08:06,346 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35172.19 MB 2025-02-14 16:08:06,553 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 16:08:06,553 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 16:08:06,553 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 16:08:06,553 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:08:06,553 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33754.76 MB 2025-02-14 16:08:06,553 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35996.62 MB 2025-02-14 16:08:06,553 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 16:08:06,553 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40179.34 MB 2025-02-14 16:08:06,553 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44426.07 MB 2025-02-14 16:08:06,553 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4246.73 MB 2025-02-14 16:08:06,553 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41540.90 MB 2025-02-14 16:08:06,554 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 16:08:06,554 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 16:08:06,554 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 16:08:06,554 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:08:06,554 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31865.23 MB 2025-02-14 16:08:06,554 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35996.62 MB 2025-02-14 16:08:06,554 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 16:08:06,554 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40179.34 MB 2025-02-14 16:08:06,554 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44426.07 MB 2025-02-14 16:08:06,554 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4246.73 MB 2025-02-14 16:08:06,554 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41540.90 MB 2025-02-14 16:08:06,713 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 16:08:06,713 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 16:08:06,713 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 16:08:06,713 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:08:06,713 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36704.40 MB 2025-02-14 16:08:06,713 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37471.41 MB 2025-02-14 16:08:06,713 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 16:08:06,713 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44426.07 MB 2025-02-14 16:08:06,713 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44839.21 MB 2025-02-14 16:08:06,713 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 16:08:06,713 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38179.19 MB 2025-02-14 16:08:06,732 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 16:08:06,732 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 16:08:06,732 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 16:08:06,732 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:08:06,732 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37884.29 MB 2025-02-14 16:08:06,732 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38089.86 MB 2025-02-14 16:08:06,732 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.56 MB 2025-02-14 16:08:06,732 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44839.21 MB 2025-02-14 16:08:06,732 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44839.21 MB 2025-02-14 16:08:06,732 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:08:06,732 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38312.53 MB 2025-02-14 16:08:06,733 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 16:08:06,733 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 16:08:06,733 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.41 seconds 2025-02-14 16:08:06,734 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:08:06,734 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26402.55 MB 2025-02-14 16:08:06,734 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38289.75 MB 2025-02-14 16:08:06,734 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11887.20 MB 2025-02-14 16:08:06,734 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48398.07 MB 2025-02-14 16:08:06,734 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44839.21 MB 2025-02-14 16:08:06,734 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3558.87 MB 2025-02-14 16:08:06,734 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38312.53 MB 2025-02-14 16:08:06,996 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 16:08:06,996 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 16:08:06,996 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 16:08:06,996 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:08:06,996 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38289.75 MB 2025-02-14 16:08:06,996 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38389.63 MB 2025-02-14 16:08:06,996 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 99.88 MB 2025-02-14 16:08:06,996 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44839.21 MB 2025-02-14 16:08:06,996 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44839.21 MB 2025-02-14 16:08:06,996 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:08:06,996 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38988.89 MB 2025-02-14 16:08:07,013 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8114, cut from 8116 2025-02-14 16:08:07,013 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 16:08:07,019 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 16:08:07,020 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 16:08:07,020 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 16:08:07,020 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:08:07,020 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27664.29 MB 2025-02-14 16:08:07,020 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31834.15 MB 2025-02-14 16:08:07,020 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4169.86 MB 2025-02-14 16:08:07,020 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44839.21 MB 2025-02-14 16:08:07,020 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49010.44 MB 2025-02-14 16:08:07,020 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4171.24 MB 2025-02-14 16:08:07,020 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36003.50 MB 2025-02-14 16:08:07,172 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7906] 2025-02-14 16:08:07,173 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:08:07,173 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:08:07,174 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:08:07,174 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 16:08:07,179 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 16:08:07,180 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:08:07,180 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 16:08:07,180 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 16:08:07,181 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:08:07,181 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:08:07,181 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:08:07,181 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:08:07,187 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 16:08:07,187 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:08:07,187 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:08:07,188 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:08:07,188 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:08:07,188 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 16:08:07,188 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:08:07,188 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:08:07,189 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:08:07,189 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 16:08:07,189 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 16:08:07,189 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:08:07,189 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:08:07,195 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:08:07,195 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:08:07,196 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:08:07,196 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:08:07,197 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:08:07,197 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:08:07,203 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:08:07,203 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:09:04,442 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:09:04,443 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:09:04,448 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 16:09:04,449 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:09:04,449 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1425, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 16:09:04,450 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:09:04,450 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1425, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 16:09:26,419 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 16:09:26,419 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 16:09:26,419 - resource_logging.py:150 - __exit__ - DEBUG - Time: 21.96 seconds 2025-02-14 16:09:26,419 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:09:26,419 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32140.67 MB 2025-02-14 16:09:26,419 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37184.32 MB 2025-02-14 16:09:26,419 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5043.65 MB 2025-02-14 16:09:26,419 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55698.26 MB 2025-02-14 16:09:26,419 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44975.52 MB 2025-02-14 16:09:26,419 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10722.74 MB 2025-02-14 16:09:26,419 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46141.88 MB 2025-02-14 16:09:26,510 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 16:09:26,510 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 16:09:26,511 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 16:09:26,511 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:09:26,511 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37184.32 MB 2025-02-14 16:09:26,511 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32428.31 MB 2025-02-14 16:09:26,511 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4756.00 MB 2025-02-14 16:09:26,511 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44975.52 MB 2025-02-14 16:09:26,511 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 57478.74 MB 2025-02-14 16:09:26,511 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 12503.22 MB 2025-02-14 16:09:26,511 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 52166.02 MB 2025-02-14 16:09:28,430 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 16:09:28,430 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 16:09:28,430 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 16:09:28,430 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:09:28,430 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32428.31 MB 2025-02-14 16:09:28,430 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32959.15 MB 2025-02-14 16:09:28,430 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 16:09:28,430 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 57478.74 MB 2025-02-14 16:09:28,431 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39931.87 MB 2025-02-14 16:09:28,431 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -17546.87 MB 2025-02-14 16:09:28,431 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36938.49 MB 2025-02-14 16:09:28,444 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 16:09:28,444 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 16:09:28,444 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 16:09:28,444 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:09:28,444 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32959.15 MB 2025-02-14 16:09:28,444 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34848.69 MB 2025-02-14 16:09:28,444 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 16:09:28,444 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39931.87 MB 2025-02-14 16:09:28,444 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39931.87 MB 2025-02-14 16:09:28,444 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:09:28,444 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36266.12 MB 2025-02-14 16:09:28,653 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 16:09:28,653 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 16:09:28,653 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 16:09:28,653 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:09:28,653 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34848.69 MB 2025-02-14 16:09:28,653 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37090.54 MB 2025-02-14 16:09:28,653 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 16:09:28,653 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39931.87 MB 2025-02-14 16:09:28,653 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45594.18 MB 2025-02-14 16:09:28,653 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 16:09:28,653 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42634.83 MB 2025-02-14 16:09:28,654 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 16:09:28,654 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 16:09:28,654 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 16:09:28,654 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:09:28,654 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32959.15 MB 2025-02-14 16:09:28,654 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37090.54 MB 2025-02-14 16:09:28,654 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 16:09:28,654 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39931.87 MB 2025-02-14 16:09:28,654 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45594.18 MB 2025-02-14 16:09:28,654 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 16:09:28,654 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42634.83 MB 2025-02-14 16:09:28,812 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 16:09:28,812 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 16:09:28,812 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 16:09:28,812 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:09:28,812 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37798.33 MB 2025-02-14 16:09:28,812 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38565.33 MB 2025-02-14 16:09:28,812 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 16:09:28,812 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45594.18 MB 2025-02-14 16:09:28,812 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46009.42 MB 2025-02-14 16:09:28,812 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 16:09:28,812 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39273.12 MB 2025-02-14 16:09:28,829 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 16:09:28,829 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 16:09:28,829 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 16:09:28,829 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:09:28,829 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38978.22 MB 2025-02-14 16:09:28,829 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39184.19 MB 2025-02-14 16:09:28,829 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.97 MB 2025-02-14 16:09:28,829 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46009.42 MB 2025-02-14 16:09:28,829 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46009.42 MB 2025-02-14 16:09:28,829 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:09:28,829 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39406.90 MB 2025-02-14 16:09:28,830 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 16:09:28,830 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 16:09:28,830 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.38 seconds 2025-02-14 16:09:28,830 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:09:28,830 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27175.85 MB 2025-02-14 16:09:28,830 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39384.23 MB 2025-02-14 16:09:28,830 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12208.38 MB 2025-02-14 16:09:28,830 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55698.26 MB 2025-02-14 16:09:28,830 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46009.42 MB 2025-02-14 16:09:28,830 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9688.84 MB 2025-02-14 16:09:28,830 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39406.90 MB 2025-02-14 16:09:29,094 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 16:09:29,094 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 16:09:29,094 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 16:09:29,094 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:09:29,094 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39384.23 MB 2025-02-14 16:09:29,094 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39484.18 MB 2025-02-14 16:09:29,094 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 99.95 MB 2025-02-14 16:09:29,094 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46009.42 MB 2025-02-14 16:09:29,094 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46009.42 MB 2025-02-14 16:09:29,094 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:09:29,094 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40083.89 MB 2025-02-14 16:09:29,112 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8120, cut from 8122 2025-02-14 16:09:29,112 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-14 16:09:29,118 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 16:09:29,118 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 16:09:29,119 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 16:09:29,119 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:09:29,119 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28437.74 MB 2025-02-14 16:09:29,119 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32611.07 MB 2025-02-14 16:09:29,119 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4173.33 MB 2025-02-14 16:09:29,119 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46009.42 MB 2025-02-14 16:09:29,119 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54356.08 MB 2025-02-14 16:09:29,119 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8346.66 MB 2025-02-14 16:09:29,119 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36784.41 MB 2025-02-14 16:09:29,271 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7912] 2025-02-14 16:09:29,273 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:09:29,273 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:09:29,274 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:09:29,274 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 16:09:29,278 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 16:09:29,279 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:09:29,279 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 16:09:29,279 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-14 16:09:29,280 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:09:29,280 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:09:29,281 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:09:29,281 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:09:29,287 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 16:09:29,288 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:09:29,288 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:09:29,289 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:09:29,289 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:09:29,289 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 16:09:29,289 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:09:29,289 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:09:29,290 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:09:29,290 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 16:09:29,290 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 16:09:29,291 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:09:29,291 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:09:29,294 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:09:29,295 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:09:29,295 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:09:29,295 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:09:29,296 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:09:29,296 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:09:29,303 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:09:29,303 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:10:26,328 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:10:26,329 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:10:26,334 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 16:10:26,336 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:10:26,336 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1268, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 16:10:26,337 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:10:26,337 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1268, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 16:10:45,965 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 16:10:45,965 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 16:10:45,965 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.62 seconds 2025-02-14 16:10:45,965 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:10:45,965 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31168.44 MB 2025-02-14 16:10:45,965 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35656.34 MB 2025-02-14 16:10:45,965 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4487.91 MB 2025-02-14 16:10:45,965 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61165.54 MB 2025-02-14 16:10:45,965 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44541.41 MB 2025-02-14 16:10:45,965 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16624.12 MB 2025-02-14 16:10:45,965 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44490.18 MB 2025-02-14 16:10:46,042 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 16:10:46,042 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 16:10:46,042 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 16:10:46,042 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:10:46,042 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35656.34 MB 2025-02-14 16:10:46,042 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31733.89 MB 2025-02-14 16:10:46,042 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3922.45 MB 2025-02-14 16:10:46,042 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44541.41 MB 2025-02-14 16:10:46,042 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53731.13 MB 2025-02-14 16:10:46,042 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9189.72 MB 2025-02-14 16:10:46,042 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49054.49 MB 2025-02-14 16:10:47,956 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 16:10:47,956 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 16:10:47,956 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 16:10:47,956 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:10:47,956 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31733.89 MB 2025-02-14 16:10:47,956 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32264.73 MB 2025-02-14 16:10:47,956 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 16:10:47,956 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53731.13 MB 2025-02-14 16:10:47,956 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40053.51 MB 2025-02-14 16:10:47,956 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13677.63 MB 2025-02-14 16:10:47,956 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36244.07 MB 2025-02-14 16:10:47,970 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 16:10:47,970 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 16:10:47,970 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 16:10:47,970 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:10:47,970 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32264.73 MB 2025-02-14 16:10:47,970 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34154.27 MB 2025-02-14 16:10:47,970 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 16:10:47,970 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40053.51 MB 2025-02-14 16:10:47,970 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40053.51 MB 2025-02-14 16:10:47,970 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:10:47,970 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35571.70 MB 2025-02-14 16:10:48,186 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 16:10:48,186 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 16:10:48,186 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 16:10:48,187 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:10:48,187 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34154.27 MB 2025-02-14 16:10:48,187 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36396.12 MB 2025-02-14 16:10:48,187 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 16:10:48,187 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40053.51 MB 2025-02-14 16:10:48,187 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45243.96 MB 2025-02-14 16:10:48,187 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 16:10:48,187 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41940.41 MB 2025-02-14 16:10:48,187 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 16:10:48,187 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 16:10:48,187 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 16:10:48,187 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:10:48,187 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32264.73 MB 2025-02-14 16:10:48,187 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36396.12 MB 2025-02-14 16:10:48,187 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 16:10:48,187 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40053.51 MB 2025-02-14 16:10:48,187 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45243.96 MB 2025-02-14 16:10:48,187 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 16:10:48,187 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41940.41 MB 2025-02-14 16:10:48,354 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 16:10:48,354 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 16:10:48,354 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 16:10:48,354 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:10:48,354 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37103.91 MB 2025-02-14 16:10:48,354 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37870.91 MB 2025-02-14 16:10:48,354 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 16:10:48,354 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45243.96 MB 2025-02-14 16:10:48,354 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45659.19 MB 2025-02-14 16:10:48,354 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 16:10:48,354 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38578.70 MB 2025-02-14 16:10:48,371 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 16:10:48,371 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 16:10:48,371 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 16:10:48,371 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:10:48,371 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38283.80 MB 2025-02-14 16:10:48,371 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38490.10 MB 2025-02-14 16:10:48,371 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.30 MB 2025-02-14 16:10:48,371 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45659.19 MB 2025-02-14 16:10:48,371 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45659.19 MB 2025-02-14 16:10:48,371 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:10:48,371 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38717.27 MB 2025-02-14 16:10:48,373 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 16:10:48,373 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 16:10:48,373 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.03 seconds 2025-02-14 16:10:48,373 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:10:48,373 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26750.62 MB 2025-02-14 16:10:48,373 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38690.88 MB 2025-02-14 16:10:48,373 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11940.26 MB 2025-02-14 16:10:48,373 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61165.54 MB 2025-02-14 16:10:48,373 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45659.19 MB 2025-02-14 16:10:48,373 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15506.34 MB 2025-02-14 16:10:48,373 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38717.27 MB 2025-02-14 16:10:48,637 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 16:10:48,637 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 16:10:48,637 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 16:10:48,637 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:10:48,638 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38690.88 MB 2025-02-14 16:10:48,638 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38791.20 MB 2025-02-14 16:10:48,638 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.32 MB 2025-02-14 16:10:48,638 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45659.19 MB 2025-02-14 16:10:48,638 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45659.19 MB 2025-02-14 16:10:48,638 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:10:48,638 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39393.12 MB 2025-02-14 16:10:48,655 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8150, cut from 8152 2025-02-14 16:10:48,656 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 16:10:48,662 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 16:10:48,662 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 16:10:48,662 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 16:10:48,662 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:10:48,662 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28013.25 MB 2025-02-14 16:10:48,662 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32201.58 MB 2025-02-14 16:10:48,662 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4188.33 MB 2025-02-14 16:10:48,662 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45659.19 MB 2025-02-14 16:10:48,662 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54037.32 MB 2025-02-14 16:10:48,662 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8378.12 MB 2025-02-14 16:10:48,662 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36389.59 MB 2025-02-14 16:10:48,823 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7942] 2025-02-14 16:10:48,824 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:10:48,824 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:10:48,825 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:10:48,825 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 16:10:48,830 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 16:10:48,831 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:10:48,831 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 16:10:48,831 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 16:10:48,832 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:10:48,832 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:10:48,832 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:10:48,832 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:10:48,838 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 16:10:48,839 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:10:48,839 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:10:48,839 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:10:48,839 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:10:48,839 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 16:10:48,840 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:10:48,840 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:10:48,840 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:10:48,840 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 16:10:48,840 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 16:10:48,841 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:10:48,841 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:10:48,844 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:10:48,844 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:10:48,845 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:10:48,845 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:10:48,846 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:10:48,846 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:10:48,852 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:10:48,852 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:10:56,126 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:10:56,127 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:10:56,134 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 16:10:56,136 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:10:56,137 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1283, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 16:10:56,138 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:10:56,138 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1283, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 16:11:16,141 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 16:11:16,141 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 16:11:16,141 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.99 seconds 2025-02-14 16:11:16,141 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:11:16,141 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31394.74 MB 2025-02-14 16:11:16,141 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35935.20 MB 2025-02-14 16:11:16,141 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4540.47 MB 2025-02-14 16:11:16,141 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60968.40 MB 2025-02-14 16:11:16,141 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46529.51 MB 2025-02-14 16:11:16,141 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -14438.89 MB 2025-02-14 16:11:16,141 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44942.97 MB 2025-02-14 16:11:16,214 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 16:11:16,214 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 16:11:16,214 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 16:11:16,214 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:11:16,214 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35935.20 MB 2025-02-14 16:11:16,214 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31933.65 MB 2025-02-14 16:11:16,214 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -4001.55 MB 2025-02-14 16:11:16,214 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46529.51 MB 2025-02-14 16:11:16,214 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55543.07 MB 2025-02-14 16:11:16,214 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 9013.56 MB 2025-02-14 16:11:16,214 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49499.03 MB 2025-02-14 16:11:18,140 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 16:11:18,140 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 16:11:18,140 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 16:11:18,141 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:11:18,141 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31933.65 MB 2025-02-14 16:11:18,141 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32464.49 MB 2025-02-14 16:11:18,141 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 16:11:18,141 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55543.07 MB 2025-02-14 16:11:18,141 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41987.08 MB 2025-02-14 16:11:18,141 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13555.99 MB 2025-02-14 16:11:18,141 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36443.82 MB 2025-02-14 16:11:18,154 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 16:11:18,154 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 16:11:18,154 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 16:11:18,154 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:11:18,154 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32464.49 MB 2025-02-14 16:11:18,154 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34354.02 MB 2025-02-14 16:11:18,154 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 16:11:18,154 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41987.08 MB 2025-02-14 16:11:18,154 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 41987.08 MB 2025-02-14 16:11:18,154 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:11:18,154 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35771.45 MB 2025-02-14 16:11:18,360 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 16:11:18,360 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 16:11:18,360 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.20 seconds 2025-02-14 16:11:18,360 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:11:18,360 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34354.02 MB 2025-02-14 16:11:18,360 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36595.88 MB 2025-02-14 16:11:18,360 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 16:11:18,360 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41987.08 MB 2025-02-14 16:11:18,360 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44818.24 MB 2025-02-14 16:11:18,360 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-14 16:11:18,360 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42140.16 MB 2025-02-14 16:11:18,360 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 16:11:18,360 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 16:11:18,360 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 16:11:18,360 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:11:18,360 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32464.49 MB 2025-02-14 16:11:18,360 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36595.88 MB 2025-02-14 16:11:18,360 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 16:11:18,361 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 41987.08 MB 2025-02-14 16:11:18,361 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44818.24 MB 2025-02-14 16:11:18,361 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2831.16 MB 2025-02-14 16:11:18,361 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42140.16 MB 2025-02-14 16:11:18,515 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 16:11:18,515 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 16:11:18,515 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 16:11:18,515 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:11:18,515 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37303.67 MB 2025-02-14 16:11:18,515 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38070.67 MB 2025-02-14 16:11:18,515 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 16:11:18,515 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44818.24 MB 2025-02-14 16:11:18,515 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45231.37 MB 2025-02-14 16:11:18,515 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 16:11:18,515 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38778.46 MB 2025-02-14 16:11:18,532 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 16:11:18,532 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 16:11:18,532 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 16:11:18,532 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:11:18,532 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38483.56 MB 2025-02-14 16:11:18,532 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38690.27 MB 2025-02-14 16:11:18,532 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.71 MB 2025-02-14 16:11:18,532 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45231.37 MB 2025-02-14 16:11:18,532 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45231.37 MB 2025-02-14 16:11:18,532 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:11:18,532 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38907.83 MB 2025-02-14 16:11:18,533 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 16:11:18,533 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 16:11:18,533 - resource_logging.py:150 - __exit__ - DEBUG - Time: 22.39 seconds 2025-02-14 16:11:18,533 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:11:18,533 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26924.66 MB 2025-02-14 16:11:18,533 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38891.29 MB 2025-02-14 16:11:18,533 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11966.63 MB 2025-02-14 16:11:18,533 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60968.40 MB 2025-02-14 16:11:18,533 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45231.37 MB 2025-02-14 16:11:18,533 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15737.03 MB 2025-02-14 16:11:18,533 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38907.83 MB 2025-02-14 16:11:18,798 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 16:11:18,798 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 16:11:18,798 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 16:11:18,798 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:11:18,798 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38891.29 MB 2025-02-14 16:11:18,798 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38991.73 MB 2025-02-14 16:11:18,798 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.44 MB 2025-02-14 16:11:18,798 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45231.37 MB 2025-02-14 16:11:18,798 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45231.37 MB 2025-02-14 16:11:18,798 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:11:18,798 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39594.38 MB 2025-02-14 16:11:18,816 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8160, cut from 8162 2025-02-14 16:11:18,817 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 16:11:18,823 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 16:11:18,823 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 16:11:18,823 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 16:11:18,823 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:11:18,823 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28187.53 MB 2025-02-14 16:11:18,823 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32381.84 MB 2025-02-14 16:11:18,823 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4194.31 MB 2025-02-14 16:11:18,823 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45231.37 MB 2025-02-14 16:11:18,823 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53619.98 MB 2025-02-14 16:11:18,823 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8388.61 MB 2025-02-14 16:11:18,823 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36574.78 MB 2025-02-14 16:11:18,973 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7952] 2025-02-14 16:11:18,975 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:11:18,975 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:11:18,976 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:11:18,976 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 16:11:18,980 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 16:11:18,981 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:11:18,981 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 16:11:18,981 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 16:11:18,982 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:11:18,982 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:11:18,982 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:11:18,982 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:11:18,988 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 16:11:18,988 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:11:18,988 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:11:18,989 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:11:18,989 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:11:18,989 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 16:11:18,989 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:11:18,989 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:11:18,990 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:11:18,990 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 16:11:18,990 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 16:11:18,990 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:11:18,990 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:11:18,994 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:11:18,994 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:11:18,995 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:11:18,995 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:11:18,996 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:11:18,996 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:11:19,001 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:11:19,001 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:12:00,398 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:12:00,398 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:12:00,403 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 16:12:00,404 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:12:00,404 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 100, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 16:12:00,405 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:12:00,405 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 100, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 16:12:01,975 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 16:12:01,975 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 16:12:01,975 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.57 seconds 2025-02-14 16:12:01,975 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:12:01,975 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23273.18 MB 2025-02-14 16:12:01,975 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23627.07 MB 2025-02-14 16:12:01,975 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 353.89 MB 2025-02-14 16:12:01,975 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60672.70 MB 2025-02-14 16:12:01,975 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25851.59 MB 2025-02-14 16:12:01,975 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -34821.11 MB 2025-02-14 16:12:01,975 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32518.05 MB 2025-02-14 16:12:01,979 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 16:12:01,979 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 16:12:01,979 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 16:12:01,979 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:12:01,979 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23627.07 MB 2025-02-14 16:12:01,979 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23798.53 MB 2025-02-14 16:12:01,979 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 171.46 MB 2025-02-14 16:12:01,979 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25851.59 MB 2025-02-14 16:12:01,979 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25851.59 MB 2025-02-14 16:12:01,979 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:12:01,979 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24329.44 MB 2025-02-14 16:12:02,473 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 16:12:02,473 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 16:12:02,473 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.49 seconds 2025-02-14 16:12:02,473 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:12:02,473 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23798.53 MB 2025-02-14 16:12:02,473 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23931.24 MB 2025-02-14 16:12:02,473 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 132.71 MB 2025-02-14 16:12:02,473 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25851.59 MB 2025-02-14 16:12:02,473 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25379.73 MB 2025-02-14 16:12:02,473 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -471.86 MB 2025-02-14 16:12:02,473 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27885.07 MB 2025-02-14 16:12:02,479 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 16:12:02,480 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 16:12:02,480 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 16:12:02,480 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:12:02,480 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23931.18 MB 2025-02-14 16:12:02,480 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24405.54 MB 2025-02-14 16:12:02,480 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 474.37 MB 2025-02-14 16:12:02,480 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25379.73 MB 2025-02-14 16:12:02,480 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 25853.69 MB 2025-02-14 16:12:02,480 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 473.96 MB 2025-02-14 16:12:02,480 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 24759.91 MB 2025-02-14 16:12:02,582 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 16:12:02,582 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 16:12:02,582 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 16:12:02,582 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:12:02,582 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24405.54 MB 2025-02-14 16:12:02,582 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24980.21 MB 2025-02-14 16:12:02,582 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 574.67 MB 2025-02-14 16:12:02,582 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25853.69 MB 2025-02-14 16:12:02,582 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27275.56 MB 2025-02-14 16:12:02,582 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1421.87 MB 2025-02-14 16:12:02,582 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26357.32 MB 2025-02-14 16:12:02,582 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 16:12:02,582 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 16:12:02,582 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 16:12:02,582 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:12:02,582 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23931.24 MB 2025-02-14 16:12:02,582 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24980.21 MB 2025-02-14 16:12:02,582 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1048.97 MB 2025-02-14 16:12:02,583 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 25379.73 MB 2025-02-14 16:12:02,583 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27275.56 MB 2025-02-14 16:12:02,583 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1895.83 MB 2025-02-14 16:12:02,583 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26357.32 MB 2025-02-14 16:12:02,633 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 16:12:02,634 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 16:12:02,634 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-14 16:12:02,634 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:12:02,634 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25235.80 MB 2025-02-14 16:12:02,634 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25476.70 MB 2025-02-14 16:12:02,634 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 240.90 MB 2025-02-14 16:12:02,634 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27275.56 MB 2025-02-14 16:12:02,634 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27426.55 MB 2025-02-14 16:12:02,634 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 150.99 MB 2025-02-14 16:12:02,634 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25653.65 MB 2025-02-14 16:12:02,639 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 16:12:02,639 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 16:12:02,639 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.00 seconds 2025-02-14 16:12:02,639 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:12:02,639 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25630.11 MB 2025-02-14 16:12:02,639 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25829.75 MB 2025-02-14 16:12:02,639 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 199.64 MB 2025-02-14 16:12:02,639 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27426.55 MB 2025-02-14 16:12:02,639 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27426.55 MB 2025-02-14 16:12:02,640 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:12:02,640 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 25829.75 MB 2025-02-14 16:12:02,641 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 16:12:02,641 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 16:12:02,641 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.23 seconds 2025-02-14 16:12:02,641 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:12:02,641 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 22924.77 MB 2025-02-14 16:12:02,641 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26030.26 MB 2025-02-14 16:12:02,641 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3105.49 MB 2025-02-14 16:12:02,641 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60672.70 MB 2025-02-14 16:12:02,641 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27426.55 MB 2025-02-14 16:12:02,641 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -33246.15 MB 2025-02-14 16:12:02,641 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26030.26 MB 2025-02-14 16:12:02,903 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 16:12:02,903 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 16:12:02,903 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 16:12:02,903 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:12:02,903 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23290.50 MB 2025-02-14 16:12:02,903 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 23390.69 MB 2025-02-14 16:12:02,903 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.18 MB 2025-02-14 16:12:02,903 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27426.55 MB 2025-02-14 16:12:02,903 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27426.55 MB 2025-02-14 16:12:02,903 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:12:02,903 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 23991.79 MB 2025-02-14 16:12:02,921 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8139, cut from 8141 2025-02-14 16:12:02,922 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 16:12:02,928 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 16:12:02,928 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 16:12:02,928 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 16:12:02,928 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:12:02,928 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23390.69 MB 2025-02-14 16:12:02,928 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27573.37 MB 2025-02-14 16:12:02,928 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4182.69 MB 2025-02-14 16:12:02,928 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27426.55 MB 2025-02-14 16:12:02,928 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37887.15 MB 2025-02-14 16:12:02,928 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10460.59 MB 2025-02-14 16:12:02,928 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31755.55 MB 2025-02-14 16:12:03,100 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7931] 2025-02-14 16:12:03,102 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:12:03,102 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:12:03,103 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:12:03,103 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 16:12:03,109 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 16:12:03,110 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:12:03,110 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 16:12:03,110 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 16:12:03,111 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:12:03,111 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:12:03,111 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:12:03,111 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:12:03,117 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 16:12:03,118 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:12:03,118 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:12:03,118 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:12:03,118 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:12:03,119 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 16:12:03,119 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:12:03,119 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:12:03,119 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:12:03,119 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 16:12:03,120 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 16:12:03,120 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:12:03,120 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:12:03,126 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:12:03,126 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:12:03,128 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:12:03,128 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:12:03,130 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:12:03,130 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:12:03,137 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:12:03,138 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:12:58,330 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:12:58,330 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:12:58,336 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 16:12:58,337 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:12:58,337 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 843, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 16:12:58,338 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:12:58,338 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 843, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 16:13:11,298 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 16:13:11,299 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 16:13:11,299 - resource_logging.py:150 - __exit__ - DEBUG - Time: 12.95 seconds 2025-02-14 16:13:11,299 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:13:11,299 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28572.52 MB 2025-02-14 16:13:11,299 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31556.76 MB 2025-02-14 16:13:11,299 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2984.25 MB 2025-02-14 16:13:11,299 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45061.51 MB 2025-02-14 16:13:11,299 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39185.29 MB 2025-02-14 16:13:11,299 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -5876.22 MB 2025-02-14 16:13:11,299 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40535.30 MB 2025-02-14 16:13:11,351 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 16:13:11,351 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 16:13:11,351 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.05 seconds 2025-02-14 16:13:11,351 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:13:11,351 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31556.76 MB 2025-02-14 16:13:11,351 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29890.00 MB 2025-02-14 16:13:11,351 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -1666.77 MB 2025-02-14 16:13:11,351 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39185.29 MB 2025-02-14 16:13:11,351 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46032.49 MB 2025-02-14 16:13:11,351 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6847.20 MB 2025-02-14 16:13:11,351 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41186.83 MB 2025-02-14 16:13:13,264 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 16:13:13,264 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 16:13:13,264 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.91 seconds 2025-02-14 16:13:13,264 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:13:13,265 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29890.00 MB 2025-02-14 16:13:13,265 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30420.84 MB 2025-02-14 16:13:13,265 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 16:13:13,265 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46032.49 MB 2025-02-14 16:13:13,265 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37616.62 MB 2025-02-14 16:13:13,265 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8415.87 MB 2025-02-14 16:13:13,265 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34400.17 MB 2025-02-14 16:13:13,278 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 16:13:13,278 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 16:13:13,278 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 16:13:13,278 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:13:13,278 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30420.84 MB 2025-02-14 16:13:13,278 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32310.37 MB 2025-02-14 16:13:13,278 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 16:13:13,278 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37616.62 MB 2025-02-14 16:13:13,278 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37616.62 MB 2025-02-14 16:13:13,278 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:13:13,278 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33727.80 MB 2025-02-14 16:13:13,495 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 16:13:13,495 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 16:13:13,495 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 16:13:13,495 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:13:13,495 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32310.37 MB 2025-02-14 16:13:13,495 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34552.23 MB 2025-02-14 16:13:13,495 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 16:13:13,495 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37616.62 MB 2025-02-14 16:13:13,495 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42807.07 MB 2025-02-14 16:13:13,495 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 16:13:13,495 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40096.51 MB 2025-02-14 16:13:13,496 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 16:13:13,496 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 16:13:13,496 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 16:13:13,496 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:13:13,496 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30420.84 MB 2025-02-14 16:13:13,496 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34552.23 MB 2025-02-14 16:13:13,496 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 16:13:13,496 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37616.62 MB 2025-02-14 16:13:13,496 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 42807.07 MB 2025-02-14 16:13:13,496 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 16:13:13,496 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40096.51 MB 2025-02-14 16:13:13,665 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 16:13:13,665 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 16:13:13,665 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 16:13:13,665 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:13:13,665 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35260.02 MB 2025-02-14 16:13:13,665 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36027.02 MB 2025-02-14 16:13:13,665 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 16:13:13,665 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 42807.07 MB 2025-02-14 16:13:13,666 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43218.11 MB 2025-02-14 16:13:13,666 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 411.04 MB 2025-02-14 16:13:13,666 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36734.81 MB 2025-02-14 16:13:13,683 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 16:13:13,683 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 16:13:13,683 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 16:13:13,683 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:13:13,683 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36439.91 MB 2025-02-14 16:13:13,683 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36646.04 MB 2025-02-14 16:13:13,683 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.13 MB 2025-02-14 16:13:13,683 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43218.11 MB 2025-02-14 16:13:13,683 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43218.11 MB 2025-02-14 16:13:13,683 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:13:13,683 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36838.31 MB 2025-02-14 16:13:13,685 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 16:13:13,685 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 16:13:13,685 - resource_logging.py:150 - __exit__ - DEBUG - Time: 15.34 seconds 2025-02-14 16:13:13,685 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:13:13,685 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25635.44 MB 2025-02-14 16:13:13,685 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36846.10 MB 2025-02-14 16:13:13,685 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11210.66 MB 2025-02-14 16:13:13,685 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45061.51 MB 2025-02-14 16:13:13,685 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43218.11 MB 2025-02-14 16:13:13,685 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1843.40 MB 2025-02-14 16:13:13,685 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36846.10 MB 2025-02-14 16:13:13,951 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 16:13:13,951 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 16:13:13,951 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 16:13:13,951 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:13:13,951 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36846.10 MB 2025-02-14 16:13:13,951 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36946.07 MB 2025-02-14 16:13:13,951 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 99.96 MB 2025-02-14 16:13:13,951 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43218.11 MB 2025-02-14 16:13:13,951 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43218.11 MB 2025-02-14 16:13:13,951 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:13:13,951 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37545.84 MB 2025-02-14 16:13:13,969 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8121, cut from 8123 2025-02-14 16:13:13,969 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 16:13:13,976 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 16:13:13,976 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 16:13:13,976 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 16:13:13,976 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:13:13,976 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26897.35 MB 2025-02-14 16:13:13,976 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31070.80 MB 2025-02-14 16:13:13,976 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4173.45 MB 2025-02-14 16:13:13,976 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43218.11 MB 2025-02-14 16:13:13,976 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51566.87 MB 2025-02-14 16:13:13,976 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8348.76 MB 2025-02-14 16:13:13,976 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35244.13 MB 2025-02-14 16:13:14,147 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7913] 2025-02-14 16:13:14,149 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:13:14,149 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:13:14,150 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:13:14,150 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 16:13:14,155 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 16:13:14,156 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:13:14,156 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 16:13:14,156 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 16:13:14,157 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:13:14,157 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:13:14,157 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:13:14,157 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:13:14,163 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 16:13:14,164 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:13:14,164 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:13:14,164 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:13:14,164 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:13:14,164 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 16:13:14,165 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:13:14,165 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:13:14,165 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:13:14,165 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 16:13:14,165 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 16:13:14,166 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:13:14,166 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:13:14,172 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:13:14,172 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:13:14,174 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:13:14,174 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:13:14,175 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:13:14,175 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:13:14,182 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:13:14,182 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:13:22,288 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:13:22,288 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:13:22,293 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 16:13:22,294 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:13:22,294 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1014, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 16:13:22,295 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:13:22,295 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1014, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 16:13:38,023 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 16:13:38,023 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 16:13:38,023 - resource_logging.py:150 - __exit__ - DEBUG - Time: 15.72 seconds 2025-02-14 16:13:38,023 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:13:38,023 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29885.62 MB 2025-02-14 16:13:38,023 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33474.11 MB 2025-02-14 16:13:38,023 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3588.49 MB 2025-02-14 16:13:38,023 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58862.86 MB 2025-02-14 16:13:38,023 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40145.78 MB 2025-02-14 16:13:38,023 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -18717.08 MB 2025-02-14 16:13:38,023 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42302.20 MB 2025-02-14 16:13:38,090 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 16:13:38,090 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 16:13:38,090 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 16:13:38,090 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:13:38,090 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33474.11 MB 2025-02-14 16:13:38,090 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30900.52 MB 2025-02-14 16:13:38,090 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2573.59 MB 2025-02-14 16:13:38,090 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40145.78 MB 2025-02-14 16:13:38,090 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48666.51 MB 2025-02-14 16:13:38,090 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8520.73 MB 2025-02-14 16:13:38,090 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44504.81 MB 2025-02-14 16:13:40,014 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 16:13:40,014 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 16:13:40,014 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 16:13:40,014 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:13:40,014 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30900.52 MB 2025-02-14 16:13:40,014 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31431.36 MB 2025-02-14 16:13:40,014 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 16:13:40,014 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48666.51 MB 2025-02-14 16:13:40,014 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35878.08 MB 2025-02-14 16:13:40,014 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12788.43 MB 2025-02-14 16:13:40,014 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35411.74 MB 2025-02-14 16:13:40,028 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 16:13:40,028 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 16:13:40,028 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 16:13:40,028 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:13:40,028 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31431.36 MB 2025-02-14 16:13:40,028 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33320.90 MB 2025-02-14 16:13:40,028 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 16:13:40,028 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35878.08 MB 2025-02-14 16:13:40,028 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37765.51 MB 2025-02-14 16:13:40,028 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 16:13:40,028 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34738.77 MB 2025-02-14 16:13:40,238 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 16:13:40,238 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 16:13:40,238 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 16:13:40,238 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:13:40,238 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33320.90 MB 2025-02-14 16:13:40,238 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35563.19 MB 2025-02-14 16:13:40,238 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2242.30 MB 2025-02-14 16:13:40,238 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37765.51 MB 2025-02-14 16:13:40,238 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43427.82 MB 2025-02-14 16:13:40,238 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 16:13:40,238 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41107.48 MB 2025-02-14 16:13:40,239 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 16:13:40,239 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 16:13:40,239 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 16:13:40,239 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:13:40,239 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31431.36 MB 2025-02-14 16:13:40,239 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35563.19 MB 2025-02-14 16:13:40,239 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.83 MB 2025-02-14 16:13:40,239 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35878.08 MB 2025-02-14 16:13:40,239 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43427.82 MB 2025-02-14 16:13:40,239 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7549.75 MB 2025-02-14 16:13:40,239 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41107.48 MB 2025-02-14 16:13:40,403 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 16:13:40,403 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 16:13:40,403 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 16:13:40,403 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:13:40,403 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36270.98 MB 2025-02-14 16:13:40,403 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37037.98 MB 2025-02-14 16:13:40,403 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 16:13:40,403 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43427.82 MB 2025-02-14 16:13:40,403 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43840.96 MB 2025-02-14 16:13:40,403 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 16:13:40,403 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37745.77 MB 2025-02-14 16:13:40,420 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 16:13:40,420 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 16:13:40,420 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 16:13:40,420 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:13:40,420 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37450.87 MB 2025-02-14 16:13:40,420 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37656.79 MB 2025-02-14 16:13:40,420 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.92 MB 2025-02-14 16:13:40,420 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43840.96 MB 2025-02-14 16:13:40,420 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43840.96 MB 2025-02-14 16:13:40,420 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:13:40,420 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37856.61 MB 2025-02-14 16:13:40,422 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 16:13:40,422 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 16:13:40,422 - resource_logging.py:150 - __exit__ - DEBUG - Time: 18.12 seconds 2025-02-14 16:13:40,422 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:13:40,422 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26352.76 MB 2025-02-14 16:13:40,422 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37857.87 MB 2025-02-14 16:13:40,422 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11505.10 MB 2025-02-14 16:13:40,422 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 58862.86 MB 2025-02-14 16:13:40,422 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43840.96 MB 2025-02-14 16:13:40,422 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -15021.90 MB 2025-02-14 16:13:40,422 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37857.87 MB 2025-02-14 16:13:40,687 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 16:13:40,687 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 16:13:40,687 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 16:13:40,687 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:13:40,687 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37857.87 MB 2025-02-14 16:13:40,687 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37958.33 MB 2025-02-14 16:13:40,687 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.47 MB 2025-02-14 16:13:40,687 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43840.96 MB 2025-02-14 16:13:40,687 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43840.96 MB 2025-02-14 16:13:40,687 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:13:40,687 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38561.13 MB 2025-02-14 16:13:40,706 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 16:13:40,707 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 16:13:40,713 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 16:13:40,713 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 16:13:40,713 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 16:13:40,713 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:13:40,713 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27615.69 MB 2025-02-14 16:13:40,713 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31810.17 MB 2025-02-14 16:13:40,713 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4194.49 MB 2025-02-14 16:13:40,713 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43840.96 MB 2025-02-14 16:13:40,713 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52231.67 MB 2025-02-14 16:13:40,713 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 16:13:40,713 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36004.48 MB 2025-02-14 16:13:40,877 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 16:13:40,878 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:13:40,878 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:13:40,879 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:13:40,879 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 16:13:40,884 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 16:13:40,885 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:13:40,885 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 16:13:40,885 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 16:13:40,886 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:13:40,886 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:13:40,886 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:13:40,887 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:13:40,892 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 16:13:40,893 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:13:40,893 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:13:40,893 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:13:40,893 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:13:40,893 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 16:13:40,894 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:13:40,894 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:13:40,894 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:13:40,894 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 16:13:40,895 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 16:13:40,895 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:13:40,895 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:13:40,900 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:13:40,900 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:13:40,901 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:13:40,901 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:13:40,903 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:13:40,903 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:13:40,910 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:13:40,910 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:14:21,218 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:14:21,218 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:14:21,223 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 16:14:21,224 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:14:21,224 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 185, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 16:14:21,225 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:14:21,225 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 185, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 16:14:24,158 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 16:14:24,158 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 16:14:24,158 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.93 seconds 2025-02-14 16:14:24,158 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:14:24,159 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24230.79 MB 2025-02-14 16:14:24,159 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 24885.50 MB 2025-02-14 16:14:24,159 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 654.70 MB 2025-02-14 16:14:24,159 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59649.29 MB 2025-02-14 16:14:24,159 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27344.76 MB 2025-02-14 16:14:24,159 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -32304.53 MB 2025-02-14 16:14:24,159 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33703.95 MB 2025-02-14 16:14:24,176 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 16:14:24,177 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 16:14:24,177 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 16:14:24,177 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:14:24,177 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24885.50 MB 2025-02-14 16:14:24,177 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25062.24 MB 2025-02-14 16:14:24,177 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 176.74 MB 2025-02-14 16:14:24,177 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27344.76 MB 2025-02-14 16:14:24,177 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 28514.98 MB 2025-02-14 16:14:24,177 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1170.21 MB 2025-02-14 16:14:24,177 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27235.01 MB 2025-02-14 16:14:25,002 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 16:14:25,002 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 16:14:25,002 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.82 seconds 2025-02-14 16:14:25,002 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:14:25,002 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25062.24 MB 2025-02-14 16:14:25,002 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 25281.21 MB 2025-02-14 16:14:25,002 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 218.97 MB 2025-02-14 16:14:25,002 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 28514.98 MB 2025-02-14 16:14:25,002 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 26801.60 MB 2025-02-14 16:14:25,002 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -1713.37 MB 2025-02-14 16:14:25,002 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29234.75 MB 2025-02-14 16:14:25,010 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 16:14:25,010 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 16:14:25,010 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 16:14:25,010 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:14:25,010 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25281.15 MB 2025-02-14 16:14:25,010 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26061.96 MB 2025-02-14 16:14:25,010 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 780.82 MB 2025-02-14 16:14:25,011 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26801.60 MB 2025-02-14 16:14:25,011 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 27971.81 MB 2025-02-14 16:14:25,011 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1170.21 MB 2025-02-14 16:14:25,011 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 26647.44 MB 2025-02-14 16:14:25,098 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 16:14:25,098 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 16:14:25,098 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 16:14:25,098 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:14:25,098 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26061.96 MB 2025-02-14 16:14:25,098 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26987.55 MB 2025-02-14 16:14:25,098 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 925.59 MB 2025-02-14 16:14:25,098 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 27971.81 MB 2025-02-14 16:14:25,098 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30312.24 MB 2025-02-14 16:14:25,098 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2340.42 MB 2025-02-14 16:14:25,098 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29278.46 MB 2025-02-14 16:14:25,099 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 16:14:25,099 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 16:14:25,099 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.09 seconds 2025-02-14 16:14:25,099 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:14:25,099 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25281.15 MB 2025-02-14 16:14:25,099 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26987.55 MB 2025-02-14 16:14:25,099 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1706.41 MB 2025-02-14 16:14:25,099 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 26801.60 MB 2025-02-14 16:14:25,099 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30312.24 MB 2025-02-14 16:14:25,099 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 3510.63 MB 2025-02-14 16:14:25,099 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29278.46 MB 2025-02-14 16:14:25,164 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 16:14:25,164 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 16:14:25,164 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 16:14:25,164 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:14:25,164 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27279.52 MB 2025-02-14 16:14:25,164 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27596.69 MB 2025-02-14 16:14:25,164 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 317.17 MB 2025-02-14 16:14:25,164 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30312.24 MB 2025-02-14 16:14:25,164 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30480.01 MB 2025-02-14 16:14:25,164 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 167.77 MB 2025-02-14 16:14:25,164 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27896.41 MB 2025-02-14 16:14:25,172 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 16:14:25,172 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 16:14:25,172 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 16:14:25,172 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:14:25,173 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27767.01 MB 2025-02-14 16:14:25,173 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27972.22 MB 2025-02-14 16:14:25,173 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.21 MB 2025-02-14 16:14:25,173 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30480.01 MB 2025-02-14 16:14:25,173 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30480.01 MB 2025-02-14 16:14:25,173 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:14:25,173 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 27978.92 MB 2025-02-14 16:14:25,174 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 16:14:25,174 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 16:14:25,174 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.95 seconds 2025-02-14 16:14:25,174 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:14:25,174 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 23586.24 MB 2025-02-14 16:14:25,174 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28172.85 MB 2025-02-14 16:14:25,174 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4586.61 MB 2025-02-14 16:14:25,174 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59649.29 MB 2025-02-14 16:14:25,174 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30480.01 MB 2025-02-14 16:14:25,174 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -29169.29 MB 2025-02-14 16:14:25,174 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28172.85 MB 2025-02-14 16:14:25,435 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 16:14:25,436 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 16:14:25,436 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 16:14:25,436 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:14:25,436 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28172.85 MB 2025-02-14 16:14:25,436 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28273.10 MB 2025-02-14 16:14:25,436 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.25 MB 2025-02-14 16:14:25,436 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30480.01 MB 2025-02-14 16:14:25,436 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 30480.01 MB 2025-02-14 16:14:25,436 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:14:25,436 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28874.57 MB 2025-02-14 16:14:25,454 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8144, cut from 8146 2025-02-14 16:14:25,454 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 16:14:25,460 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 16:14:25,460 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 16:14:25,460 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 16:14:25,460 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:14:25,460 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24225.63 MB 2025-02-14 16:14:25,460 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28411.54 MB 2025-02-14 16:14:25,460 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4185.92 MB 2025-02-14 16:14:25,460 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 30480.01 MB 2025-02-14 16:14:25,460 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40944.80 MB 2025-02-14 16:14:25,460 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10464.79 MB 2025-02-14 16:14:25,460 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32596.28 MB 2025-02-14 16:14:25,609 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7936] 2025-02-14 16:14:25,611 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:14:25,611 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:14:25,611 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:14:25,612 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 16:14:25,616 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 16:14:25,617 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:14:25,617 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 16:14:25,617 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 16:14:25,618 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:14:25,618 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:14:25,618 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:14:25,618 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:14:25,624 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 16:14:25,624 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:14:25,624 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:14:25,625 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:14:25,625 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:14:25,625 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 16:14:25,625 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:14:25,625 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:14:25,626 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:14:25,626 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 16:14:25,626 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 16:14:25,626 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:14:25,626 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:14:25,631 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:14:25,631 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:14:25,632 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:14:25,632 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:14:25,632 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:14:25,633 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:14:25,639 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:14:25,639 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:14:57,224 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:14:57,225 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:14:57,230 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 16:14:57,231 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:14:57,231 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 938, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 16:14:57,232 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:14:57,232 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 938, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 16:15:11,703 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 16:15:11,703 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 16:15:11,703 - resource_logging.py:150 - __exit__ - DEBUG - Time: 14.46 seconds 2025-02-14 16:15:11,703 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:15:11,703 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29600.21 MB 2025-02-14 16:15:11,703 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32920.00 MB 2025-02-14 16:15:11,703 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3319.79 MB 2025-02-14 16:15:11,703 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48484.06 MB 2025-02-14 16:15:11,703 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39650.85 MB 2025-02-14 16:15:11,703 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -8833.20 MB 2025-02-14 16:15:11,703 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41789.49 MB 2025-02-14 16:15:11,763 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 16:15:11,763 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 16:15:11,763 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 16:15:11,763 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:15:11,763 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32920.00 MB 2025-02-14 16:15:11,763 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30749.59 MB 2025-02-14 16:15:11,763 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2170.41 MB 2025-02-14 16:15:11,763 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39650.85 MB 2025-02-14 16:15:11,763 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46982.50 MB 2025-02-14 16:15:11,763 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 7331.64 MB 2025-02-14 16:15:11,763 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43093.83 MB 2025-02-14 16:15:13,684 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 16:15:13,684 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 16:15:13,684 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 16:15:13,684 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:15:13,684 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30749.59 MB 2025-02-14 16:15:13,684 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31280.43 MB 2025-02-14 16:15:13,684 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 16:15:13,684 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46982.50 MB 2025-02-14 16:15:13,684 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 37746.64 MB 2025-02-14 16:15:13,684 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9235.86 MB 2025-02-14 16:15:13,684 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35258.98 MB 2025-02-14 16:15:13,698 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 16:15:13,698 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 16:15:13,698 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 16:15:13,698 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:15:13,698 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31280.43 MB 2025-02-14 16:15:13,698 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33169.97 MB 2025-02-14 16:15:13,698 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 16:15:13,698 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37746.64 MB 2025-02-14 16:15:13,698 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38690.36 MB 2025-02-14 16:15:13,698 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 943.72 MB 2025-02-14 16:15:13,698 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34587.40 MB 2025-02-14 16:15:13,910 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 16:15:13,911 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 16:15:13,911 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 16:15:13,911 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:15:13,911 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33169.97 MB 2025-02-14 16:15:13,911 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35411.82 MB 2025-02-14 16:15:13,911 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 16:15:13,911 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38690.36 MB 2025-02-14 16:15:13,911 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43880.81 MB 2025-02-14 16:15:13,911 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 16:15:13,911 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40956.11 MB 2025-02-14 16:15:13,911 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 16:15:13,911 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 16:15:13,911 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 16:15:13,911 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:15:13,911 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31280.43 MB 2025-02-14 16:15:13,911 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35411.82 MB 2025-02-14 16:15:13,911 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 16:15:13,911 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 37746.64 MB 2025-02-14 16:15:13,911 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43880.81 MB 2025-02-14 16:15:13,911 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 16:15:13,911 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40956.11 MB 2025-02-14 16:15:14,077 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 16:15:14,077 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 16:15:14,077 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 16:15:14,077 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:15:14,077 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36119.61 MB 2025-02-14 16:15:14,077 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36886.61 MB 2025-02-14 16:15:14,077 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 16:15:14,077 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43880.81 MB 2025-02-14 16:15:14,077 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44293.95 MB 2025-02-14 16:15:14,077 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 413.14 MB 2025-02-14 16:15:14,077 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37594.40 MB 2025-02-14 16:15:14,095 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 16:15:14,095 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 16:15:14,095 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 16:15:14,095 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:15:14,095 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37299.50 MB 2025-02-14 16:15:14,095 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37506.36 MB 2025-02-14 16:15:14,095 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.86 MB 2025-02-14 16:15:14,095 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44293.95 MB 2025-02-14 16:15:14,095 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44293.95 MB 2025-02-14 16:15:14,095 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:15:14,095 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37713.75 MB 2025-02-14 16:15:14,096 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 16:15:14,096 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 16:15:14,096 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.86 seconds 2025-02-14 16:15:14,096 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:15:14,096 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26332.15 MB 2025-02-14 16:15:14,096 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37707.43 MB 2025-02-14 16:15:14,096 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11375.29 MB 2025-02-14 16:15:14,096 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48484.06 MB 2025-02-14 16:15:14,096 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44293.95 MB 2025-02-14 16:15:14,096 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -4190.11 MB 2025-02-14 16:15:14,096 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37713.75 MB 2025-02-14 16:15:14,362 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 16:15:14,362 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 16:15:14,362 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 16:15:14,362 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:15:14,362 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37707.43 MB 2025-02-14 16:15:14,362 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37807.90 MB 2025-02-14 16:15:14,362 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.47 MB 2025-02-14 16:15:14,362 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44293.95 MB 2025-02-14 16:15:14,362 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44293.95 MB 2025-02-14 16:15:14,362 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:15:14,362 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38410.70 MB 2025-02-14 16:15:14,380 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 16:15:14,381 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 16:15:14,387 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 16:15:14,387 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 16:15:14,387 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 16:15:14,387 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:15:14,387 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27595.07 MB 2025-02-14 16:15:14,387 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31789.55 MB 2025-02-14 16:15:14,387 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4194.49 MB 2025-02-14 16:15:14,387 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44293.95 MB 2025-02-14 16:15:14,387 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 52684.65 MB 2025-02-14 16:15:14,388 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 16:15:14,388 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35983.86 MB 2025-02-14 16:15:14,553 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 16:15:14,554 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:15:14,554 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:15:14,555 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:15:14,555 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 16:15:14,560 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 16:15:14,561 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:15:14,561 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 16:15:14,561 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 16:15:14,562 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:15:14,562 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:15:14,563 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:15:14,563 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:15:14,569 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 16:15:14,569 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:15:14,569 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:15:14,570 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:15:14,570 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:15:14,570 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 16:15:14,570 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:15:14,570 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:15:14,571 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:15:14,571 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 16:15:14,571 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 16:15:14,571 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:15:14,571 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:15:14,577 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:15:14,577 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:15:14,579 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:15:14,579 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:15:14,580 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:15:14,581 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:15:14,588 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:15:14,588 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:16:07,058 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:16:07,058 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:16:07,063 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 16:16:07,064 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:16:07,064 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 679, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 16:16:07,065 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:16:07,065 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 679, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 16:16:17,501 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 16:16:17,501 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 16:16:17,501 - resource_logging.py:150 - __exit__ - DEBUG - Time: 10.43 seconds 2025-02-14 16:16:17,501 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:16:17,501 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27916.61 MB 2025-02-14 16:16:17,501 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30319.55 MB 2025-02-14 16:16:17,501 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2402.94 MB 2025-02-14 16:16:17,501 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60345.55 MB 2025-02-14 16:16:17,501 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39051.07 MB 2025-02-14 16:16:17,501 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21294.48 MB 2025-02-14 16:16:17,501 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39199.92 MB 2025-02-14 16:16:17,529 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 16:16:17,529 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 16:16:17,529 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 16:16:17,529 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:16:17,529 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30319.55 MB 2025-02-14 16:16:17,529 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27986.24 MB 2025-02-14 16:16:17,529 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2333.31 MB 2025-02-14 16:16:17,529 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39051.07 MB 2025-02-14 16:16:17,529 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39051.07 MB 2025-02-14 16:16:17,529 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:16:17,529 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32886.66 MB 2025-02-14 16:16:18,392 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 16:16:18,392 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 16:16:18,392 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.86 seconds 2025-02-14 16:16:18,392 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:16:18,392 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27986.24 MB 2025-02-14 16:16:18,393 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28226.45 MB 2025-02-14 16:16:18,393 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 240.21 MB 2025-02-14 16:16:18,393 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39051.07 MB 2025-02-14 16:16:18,393 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35775.32 MB 2025-02-14 16:16:18,393 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -3275.75 MB 2025-02-14 16:16:18,393 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32156.68 MB 2025-02-14 16:16:18,401 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 16:16:18,401 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 16:16:18,401 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 16:16:18,401 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:16:18,401 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28226.45 MB 2025-02-14 16:16:18,401 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29081.25 MB 2025-02-14 16:16:18,401 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 854.81 MB 2025-02-14 16:16:18,401 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35775.32 MB 2025-02-14 16:16:18,401 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35775.32 MB 2025-02-14 16:16:18,401 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:16:18,401 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29723.23 MB 2025-02-14 16:16:18,501 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 16:16:18,501 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 16:16:18,501 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 16:16:18,501 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:16:18,502 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29081.25 MB 2025-02-14 16:16:18,502 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30095.73 MB 2025-02-14 16:16:18,502 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1014.48 MB 2025-02-14 16:16:18,502 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35775.32 MB 2025-02-14 16:16:18,502 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35775.32 MB 2025-02-14 16:16:18,502 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:16:18,502 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32605.07 MB 2025-02-14 16:16:18,502 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 16:16:18,502 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 16:16:18,502 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 16:16:18,502 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:16:18,502 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28226.45 MB 2025-02-14 16:16:18,502 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30095.73 MB 2025-02-14 16:16:18,502 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1869.28 MB 2025-02-14 16:16:18,502 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35775.32 MB 2025-02-14 16:16:18,502 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35775.32 MB 2025-02-14 16:16:18,502 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:16:18,502 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32605.07 MB 2025-02-14 16:16:18,586 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 16:16:18,586 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 16:16:18,586 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 16:16:18,586 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:16:18,586 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30416.00 MB 2025-02-14 16:16:18,586 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30763.66 MB 2025-02-14 16:16:18,586 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 347.66 MB 2025-02-14 16:16:18,586 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35775.32 MB 2025-02-14 16:16:18,586 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35959.87 MB 2025-02-14 16:16:18,586 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 184.55 MB 2025-02-14 16:16:18,586 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31089.90 MB 2025-02-14 16:16:18,596 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 16:16:18,596 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 16:16:18,596 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 16:16:18,596 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:16:18,596 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30950.50 MB 2025-02-14 16:16:18,596 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31152.14 MB 2025-02-14 16:16:18,596 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 201.64 MB 2025-02-14 16:16:18,596 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35959.87 MB 2025-02-14 16:16:18,596 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35961.96 MB 2025-02-14 16:16:18,596 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-14 16:16:18,596 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31157.75 MB 2025-02-14 16:16:18,597 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 16:16:18,597 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 16:16:18,597 - resource_logging.py:150 - __exit__ - DEBUG - Time: 11.53 seconds 2025-02-14 16:16:18,597 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:16:18,597 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25550.92 MB 2025-02-14 16:16:18,597 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31352.72 MB 2025-02-14 16:16:18,597 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5801.80 MB 2025-02-14 16:16:18,597 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 60345.55 MB 2025-02-14 16:16:18,597 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35961.96 MB 2025-02-14 16:16:18,597 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24383.59 MB 2025-02-14 16:16:18,597 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31352.72 MB 2025-02-14 16:16:18,859 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 16:16:18,859 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 16:16:18,859 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 16:16:18,859 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:16:18,859 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31352.72 MB 2025-02-14 16:16:18,859 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31452.94 MB 2025-02-14 16:16:18,859 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.22 MB 2025-02-14 16:16:18,859 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35961.96 MB 2025-02-14 16:16:18,859 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 35961.96 MB 2025-02-14 16:16:18,859 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:16:18,859 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 32054.26 MB 2025-02-14 16:16:18,877 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8142, cut from 8144 2025-02-14 16:16:18,877 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 16:16:18,883 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 16:16:18,883 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 16:16:18,883 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 16:16:18,883 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:16:18,883 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26232.54 MB 2025-02-14 16:16:18,883 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30416.77 MB 2025-02-14 16:16:18,883 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4184.22 MB 2025-02-14 16:16:18,883 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 35961.96 MB 2025-02-14 16:16:18,884 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44331.70 MB 2025-02-14 16:16:18,884 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8369.73 MB 2025-02-14 16:16:18,884 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34600.59 MB 2025-02-14 16:16:19,042 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7934] 2025-02-14 16:16:19,043 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:16:19,043 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:16:19,044 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:16:19,044 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 16:16:19,048 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 16:16:19,049 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:16:19,050 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 16:16:19,050 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 16:16:19,050 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:16:19,050 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:16:19,051 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:16:19,051 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:16:19,057 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 16:16:19,057 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:16:19,057 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:16:19,058 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:16:19,058 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:16:19,058 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 16:16:19,058 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:16:19,058 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:16:19,059 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:16:19,059 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 16:16:19,059 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 16:16:19,059 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:16:19,059 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:16:19,063 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:16:19,063 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:16:19,064 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:16:19,064 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:16:19,065 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:16:19,065 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:16:19,072 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:16:19,072 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:18:07,814 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:18:07,815 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:18:07,820 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 16:18:07,821 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:18:07,821 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1095, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 16:18:07,822 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:18:07,822 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1095, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 16:18:24,644 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 16:18:24,644 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 16:18:24,644 - resource_logging.py:150 - __exit__ - DEBUG - Time: 16.82 seconds 2025-02-14 16:18:24,644 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:18:24,644 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30937.44 MB 2025-02-14 16:18:24,644 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34812.98 MB 2025-02-14 16:18:24,644 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3875.54 MB 2025-02-14 16:18:24,644 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52114.23 MB 2025-02-14 16:18:24,644 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38352.72 MB 2025-02-14 16:18:24,644 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13761.51 MB 2025-02-14 16:18:24,644 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43807.01 MB 2025-02-14 16:18:24,717 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 16:18:24,717 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 16:18:24,717 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.07 seconds 2025-02-14 16:18:24,717 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:18:24,717 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34812.98 MB 2025-02-14 16:18:24,717 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31810.07 MB 2025-02-14 16:18:24,717 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -3002.92 MB 2025-02-14 16:18:24,717 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38352.72 MB 2025-02-14 16:18:24,717 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53527.71 MB 2025-02-14 16:18:24,717 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 15174.99 MB 2025-02-14 16:18:24,717 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46509.26 MB 2025-02-14 16:18:26,635 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 16:18:26,635 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 16:18:26,635 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 16:18:26,635 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:18:26,635 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31810.07 MB 2025-02-14 16:18:26,635 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32340.91 MB 2025-02-14 16:18:26,635 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 16:18:26,635 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53527.71 MB 2025-02-14 16:18:26,635 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 36601.59 MB 2025-02-14 16:18:26,635 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16926.11 MB 2025-02-14 16:18:26,635 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36320.78 MB 2025-02-14 16:18:26,650 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 16:18:26,650 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 16:18:26,650 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 16:18:26,650 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:18:26,650 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32340.91 MB 2025-02-14 16:18:26,650 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34230.44 MB 2025-02-14 16:18:26,650 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.53 MB 2025-02-14 16:18:26,650 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36601.59 MB 2025-02-14 16:18:26,650 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 38489.03 MB 2025-02-14 16:18:26,650 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 1887.44 MB 2025-02-14 16:18:26,650 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35647.87 MB 2025-02-14 16:18:26,865 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 16:18:26,865 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 16:18:26,865 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 16:18:26,865 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:18:26,865 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34230.44 MB 2025-02-14 16:18:26,865 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36472.30 MB 2025-02-14 16:18:26,865 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 16:18:26,865 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 38489.03 MB 2025-02-14 16:18:26,865 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44623.20 MB 2025-02-14 16:18:26,865 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 6134.17 MB 2025-02-14 16:18:26,865 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42016.58 MB 2025-02-14 16:18:26,866 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 16:18:26,866 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 16:18:26,866 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 16:18:26,866 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:18:26,866 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32340.91 MB 2025-02-14 16:18:26,866 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36472.30 MB 2025-02-14 16:18:26,866 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.39 MB 2025-02-14 16:18:26,866 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 36601.59 MB 2025-02-14 16:18:26,866 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 44623.20 MB 2025-02-14 16:18:26,866 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8021.61 MB 2025-02-14 16:18:26,866 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42016.58 MB 2025-02-14 16:18:27,033 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 16:18:27,033 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 16:18:27,033 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 16:18:27,033 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:18:27,033 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37180.09 MB 2025-02-14 16:18:27,033 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37947.09 MB 2025-02-14 16:18:27,033 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 16:18:27,033 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 44623.20 MB 2025-02-14 16:18:27,033 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45040.53 MB 2025-02-14 16:18:27,033 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 16:18:27,033 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38654.88 MB 2025-02-14 16:18:27,050 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 16:18:27,050 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 16:18:27,050 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 16:18:27,050 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:18:27,050 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38359.98 MB 2025-02-14 16:18:27,050 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38567.41 MB 2025-02-14 16:18:27,050 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 207.43 MB 2025-02-14 16:18:27,050 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45040.53 MB 2025-02-14 16:18:27,050 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45040.53 MB 2025-02-14 16:18:27,050 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:18:27,051 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38773.46 MB 2025-02-14 16:18:27,052 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 16:18:27,052 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 16:18:27,052 - resource_logging.py:150 - __exit__ - DEBUG - Time: 19.23 seconds 2025-02-14 16:18:27,052 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:18:27,052 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27122.38 MB 2025-02-14 16:18:27,052 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38768.48 MB 2025-02-14 16:18:27,052 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11646.11 MB 2025-02-14 16:18:27,052 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52114.23 MB 2025-02-14 16:18:27,052 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45040.53 MB 2025-02-14 16:18:27,052 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7073.69 MB 2025-02-14 16:18:27,052 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38773.46 MB 2025-02-14 16:18:27,314 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 16:18:27,314 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 16:18:27,314 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 16:18:27,314 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:18:27,314 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38768.48 MB 2025-02-14 16:18:27,314 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38868.95 MB 2025-02-14 16:18:27,314 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.47 MB 2025-02-14 16:18:27,314 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45040.53 MB 2025-02-14 16:18:27,314 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45040.53 MB 2025-02-14 16:18:27,314 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:18:27,314 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39471.75 MB 2025-02-14 16:18:27,332 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 16:18:27,332 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 16:18:27,338 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 16:18:27,338 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 16:18:27,338 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 16:18:27,338 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:18:27,338 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28385.30 MB 2025-02-14 16:18:27,338 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32579.78 MB 2025-02-14 16:18:27,338 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4194.49 MB 2025-02-14 16:18:27,338 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45040.53 MB 2025-02-14 16:18:27,339 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53431.24 MB 2025-02-14 16:18:27,339 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8390.71 MB 2025-02-14 16:18:27,339 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36774.09 MB 2025-02-14 16:18:27,501 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 16:18:27,502 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:18:27,502 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:18:27,503 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:18:27,503 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 16:18:27,508 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 16:18:27,509 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:18:27,509 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 16:18:27,509 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 16:18:27,510 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:18:27,510 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:18:27,510 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:18:27,510 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:18:27,516 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 16:18:27,517 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:18:27,517 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:18:27,517 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:18:27,517 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:18:27,517 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 16:18:27,518 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:18:27,518 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:18:27,518 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:18:27,518 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 16:18:27,518 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 16:18:27,519 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:18:27,519 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:18:27,525 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:18:27,525 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:18:27,527 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:18:27,527 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:18:27,529 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:18:27,529 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:18:27,537 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:18:27,537 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:20:22,089 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:20:22,089 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:20:22,097 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 16:20:22,100 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:20:22,100 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2392, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 16:20:22,101 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:20:22,102 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2392, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 16:20:58,739 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 16:20:58,739 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 16:20:58,739 - resource_logging.py:150 - __exit__ - DEBUG - Time: 36.62 seconds 2025-02-14 16:20:58,739 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:20:58,739 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40096.62 MB 2025-02-14 16:20:58,739 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 48562.82 MB 2025-02-14 16:20:58,739 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 8466.20 MB 2025-02-14 16:20:58,739 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65569.55 MB 2025-02-14 16:20:58,739 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 59276.00 MB 2025-02-14 16:20:58,739 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6293.55 MB 2025-02-14 16:20:58,739 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 57495.22 MB 2025-02-14 16:20:58,885 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 16:20:58,885 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 16:20:58,885 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.14 seconds 2025-02-14 16:20:58,885 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:20:58,885 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 48562.82 MB 2025-02-14 16:20:58,885 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38673.18 MB 2025-02-14 16:20:58,885 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -9889.64 MB 2025-02-14 16:20:58,885 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 59276.00 MB 2025-02-14 16:20:58,885 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 82208.36 MB 2025-02-14 16:20:58,885 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 22932.36 MB 2025-02-14 16:20:58,885 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 73625.59 MB 2025-02-14 16:21:00,823 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 16:21:00,823 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 16:21:00,823 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-14 16:21:00,823 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:21:00,823 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38673.18 MB 2025-02-14 16:21:00,823 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39204.03 MB 2025-02-14 16:21:00,823 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 16:21:00,823 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 82208.36 MB 2025-02-14 16:21:00,823 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46575.65 MB 2025-02-14 16:21:00,823 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -35632.71 MB 2025-02-14 16:21:00,823 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43183.36 MB 2025-02-14 16:21:00,836 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 16:21:00,836 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 16:21:00,836 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 16:21:00,836 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:21:00,836 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39204.03 MB 2025-02-14 16:21:00,836 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41093.49 MB 2025-02-14 16:21:00,836 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.47 MB 2025-02-14 16:21:00,836 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46575.65 MB 2025-02-14 16:21:00,836 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 46575.65 MB 2025-02-14 16:21:00,836 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:21:00,836 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42510.92 MB 2025-02-14 16:21:01,054 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 16:21:01,054 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 16:21:01,054 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 16:21:01,054 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:21:01,054 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41093.49 MB 2025-02-14 16:21:01,054 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43335.35 MB 2025-02-14 16:21:01,054 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 16:21:01,054 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46575.65 MB 2025-02-14 16:21:01,054 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51294.24 MB 2025-02-14 16:21:01,054 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 16:21:01,054 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48879.63 MB 2025-02-14 16:21:01,055 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 16:21:01,055 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 16:21:01,055 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 16:21:01,055 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:21:01,055 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39204.03 MB 2025-02-14 16:21:01,055 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43335.35 MB 2025-02-14 16:21:01,055 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.32 MB 2025-02-14 16:21:01,055 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 46575.65 MB 2025-02-14 16:21:01,055 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51294.24 MB 2025-02-14 16:21:01,055 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 16:21:01,055 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48879.63 MB 2025-02-14 16:21:01,220 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 16:21:01,221 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 16:21:01,221 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 16:21:01,221 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:21:01,221 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 44043.14 MB 2025-02-14 16:21:01,221 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 44810.14 MB 2025-02-14 16:21:01,221 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 16:21:01,221 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51294.24 MB 2025-02-14 16:21:01,221 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51709.48 MB 2025-02-14 16:21:01,221 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 16:21:01,221 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45517.93 MB 2025-02-14 16:21:01,238 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 16:21:01,238 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 16:21:01,238 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 16:21:01,238 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:21:01,238 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45223.03 MB 2025-02-14 16:21:01,238 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45429.58 MB 2025-02-14 16:21:01,238 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.55 MB 2025-02-14 16:21:01,238 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51709.48 MB 2025-02-14 16:21:01,238 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51709.48 MB 2025-02-14 16:21:01,238 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:21:01,238 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45662.13 MB 2025-02-14 16:21:01,239 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 16:21:01,239 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 16:21:01,239 - resource_logging.py:150 - __exit__ - DEBUG - Time: 39.13 seconds 2025-02-14 16:21:01,239 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:21:01,239 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31762.70 MB 2025-02-14 16:21:01,239 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45630.18 MB 2025-02-14 16:21:01,239 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 13867.49 MB 2025-02-14 16:21:01,239 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61335.40 MB 2025-02-14 16:21:01,239 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51709.48 MB 2025-02-14 16:21:01,239 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9625.93 MB 2025-02-14 16:21:01,239 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45662.13 MB 2025-02-14 16:21:01,504 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 16:21:01,504 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 16:21:01,504 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 16:21:01,504 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:21:01,504 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 45630.18 MB 2025-02-14 16:21:01,504 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45730.42 MB 2025-02-14 16:21:01,504 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.23 MB 2025-02-14 16:21:01,504 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51709.48 MB 2025-02-14 16:21:01,504 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 51709.48 MB 2025-02-14 16:21:01,504 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:21:01,504 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46331.82 MB 2025-02-14 16:21:01,522 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8143, cut from 8145 2025-02-14 16:21:01,522 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The engagement rate for this video is 2.'] 2025-02-14 16:21:01,528 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 16:21:01,528 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 16:21:01,528 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 16:21:01,528 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:21:01,528 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33025.15 MB 2025-02-14 16:21:01,528 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37209.89 MB 2025-02-14 16:21:01,528 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4184.74 MB 2025-02-14 16:21:01,528 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 51709.48 MB 2025-02-14 16:21:01,528 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 60081.31 MB 2025-02-14 16:21:01,528 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8371.83 MB 2025-02-14 16:21:01,528 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41394.11 MB 2025-02-14 16:21:01,695 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7935] 2025-02-14 16:21:01,697 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:21:01,697 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:21:01,698 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:21:01,698 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 16:21:01,702 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 16:21:01,703 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:21:01,703 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 16:21:01,703 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The engagement rate for this video is 2.'] 2025-02-14 16:21:01,704 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:21:01,704 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:21:01,705 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:21:01,705 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:21:01,711 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 16:21:01,711 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:21:01,711 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:21:01,712 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:21:01,712 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:21:01,712 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 16:21:01,712 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:21:01,712 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:21:01,713 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:21:01,713 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 16:21:01,713 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 16:21:01,713 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:21:01,713 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:21:01,719 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:21:01,719 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:21:01,721 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:21:01,721 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:21:01,723 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:21:01,723 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:21:01,731 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:21:01,731 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:22:19,910 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:22:19,910 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:22:19,915 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 16:22:19,917 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:22:19,917 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 2786, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 16:22:19,918 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:22:19,918 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 2786, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 16:23:02,942 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 16:23:02,942 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 16:23:02,942 - resource_logging.py:150 - __exit__ - DEBUG - Time: 43.01 seconds 2025-02-14 16:23:02,942 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:23:02,942 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42963.60 MB 2025-02-14 16:23:02,942 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 52823.09 MB 2025-02-14 16:23:02,942 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 9859.50 MB 2025-02-14 16:23:02,942 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 77661.73 MB 2025-02-14 16:23:02,942 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65638.76 MB 2025-02-14 16:23:02,942 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12022.97 MB 2025-02-14 16:23:02,942 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 62682.59 MB 2025-02-14 16:23:03,117 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 16:23:03,117 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 16:23:03,117 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.17 seconds 2025-02-14 16:23:03,117 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:23:03,117 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 52823.09 MB 2025-02-14 16:23:03,117 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40842.99 MB 2025-02-14 16:23:03,117 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -11980.10 MB 2025-02-14 16:23:03,117 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65638.76 MB 2025-02-14 16:23:03,117 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 92301.95 MB 2025-02-14 16:23:03,117 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 26663.19 MB 2025-02-14 16:23:03,117 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 82130.10 MB 2025-02-14 16:23:05,049 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 16:23:05,049 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 16:23:05,049 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 16:23:05,049 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:23:05,049 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40842.99 MB 2025-02-14 16:23:05,049 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41373.83 MB 2025-02-14 16:23:05,049 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 16:23:05,049 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 92301.95 MB 2025-02-14 16:23:05,049 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55777.95 MB 2025-02-14 16:23:05,049 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -36524.00 MB 2025-02-14 16:23:05,049 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45353.17 MB 2025-02-14 16:23:05,063 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 16:23:05,063 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 16:23:05,063 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 16:23:05,063 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:23:05,063 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41373.83 MB 2025-02-14 16:23:05,063 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43263.24 MB 2025-02-14 16:23:05,063 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.40 MB 2025-02-14 16:23:05,063 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55777.95 MB 2025-02-14 16:23:05,063 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55777.95 MB 2025-02-14 16:23:05,063 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:23:05,063 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44680.66 MB 2025-02-14 16:23:05,277 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 16:23:05,277 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 16:23:05,277 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 16:23:05,277 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:23:05,277 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 43263.24 MB 2025-02-14 16:23:05,277 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45505.09 MB 2025-02-14 16:23:05,277 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 16:23:05,277 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55777.95 MB 2025-02-14 16:23:05,277 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55777.95 MB 2025-02-14 16:23:05,277 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:23:05,277 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51049.37 MB 2025-02-14 16:23:05,278 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 16:23:05,278 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 16:23:05,278 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 16:23:05,278 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:23:05,278 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41373.83 MB 2025-02-14 16:23:05,278 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 45505.09 MB 2025-02-14 16:23:05,278 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.26 MB 2025-02-14 16:23:05,278 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55777.95 MB 2025-02-14 16:23:05,278 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 55777.95 MB 2025-02-14 16:23:05,278 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:23:05,278 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51049.37 MB 2025-02-14 16:23:05,446 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 16:23:05,447 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 16:23:05,447 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 16:23:05,447 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:23:05,447 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 46212.88 MB 2025-02-14 16:23:05,447 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 46979.88 MB 2025-02-14 16:23:05,447 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 16:23:05,447 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 55777.95 MB 2025-02-14 16:23:05,447 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56193.19 MB 2025-02-14 16:23:05,447 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 16:23:05,447 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47687.67 MB 2025-02-14 16:23:05,464 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 16:23:05,464 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 16:23:05,464 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 16:23:05,464 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:23:05,464 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47392.77 MB 2025-02-14 16:23:05,464 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47598.57 MB 2025-02-14 16:23:05,464 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.80 MB 2025-02-14 16:23:05,464 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56193.19 MB 2025-02-14 16:23:05,464 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56193.19 MB 2025-02-14 16:23:05,464 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:23:05,464 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47831.90 MB 2025-02-14 16:23:05,465 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 16:23:05,465 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 16:23:05,465 - resource_logging.py:150 - __exit__ - DEBUG - Time: 45.55 seconds 2025-02-14 16:23:05,465 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:23:05,465 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33256.53 MB 2025-02-14 16:23:05,465 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47799.17 MB 2025-02-14 16:23:05,465 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 14542.64 MB 2025-02-14 16:23:05,465 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 72884.42 MB 2025-02-14 16:23:05,465 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56193.19 MB 2025-02-14 16:23:05,465 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16691.23 MB 2025-02-14 16:23:05,465 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 47831.90 MB 2025-02-14 16:23:05,729 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 16:23:05,729 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 16:23:05,729 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 16:23:05,729 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:23:05,729 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 47799.17 MB 2025-02-14 16:23:05,729 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 47899.12 MB 2025-02-14 16:23:05,729 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 99.95 MB 2025-02-14 16:23:05,729 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56193.19 MB 2025-02-14 16:23:05,729 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56193.19 MB 2025-02-14 16:23:05,729 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:23:05,729 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 48498.83 MB 2025-02-14 16:23:05,746 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8120, cut from 8122 2025-02-14 16:23:05,746 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2.'] 2025-02-14 16:23:05,752 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 16:23:05,752 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 16:23:05,752 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 16:23:05,752 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:23:05,752 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34518.99 MB 2025-02-14 16:23:05,752 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38691.93 MB 2025-02-14 16:23:05,752 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4172.94 MB 2025-02-14 16:23:05,752 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56193.19 MB 2025-02-14 16:23:05,752 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 56193.19 MB 2025-02-14 16:23:05,752 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:23:05,752 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42864.35 MB 2025-02-14 16:23:05,918 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7912] 2025-02-14 16:23:05,919 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:23:05,919 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:23:05,920 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:23:05,920 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 16:23:05,925 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 16:23:05,926 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:23:05,926 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 16:23:05,926 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2.'] 2025-02-14 16:23:05,926 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:23:05,926 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:23:05,927 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:23:05,927 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:23:05,933 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 16:23:05,933 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:23:05,933 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:23:05,934 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:23:05,934 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:23:05,934 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 16:23:05,934 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:23:05,934 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:23:05,935 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:23:05,935 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 16:23:05,935 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 16:23:05,935 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:23:05,935 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:23:05,941 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:23:05,941 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:23:05,942 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:23:05,942 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:23:05,944 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:23:05,944 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:23:05,952 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:23:05,952 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:24:21,899 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:24:21,899 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:24:21,904 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 16:24:21,905 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:24:21,905 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1597, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 16:24:21,906 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:24:21,906 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1597, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 16:24:46,511 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 16:24:46,512 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 16:24:46,512 - resource_logging.py:150 - __exit__ - DEBUG - Time: 24.60 seconds 2025-02-14 16:24:46,512 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:24:46,512 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34799.35 MB 2025-02-14 16:24:46,512 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40451.18 MB 2025-02-14 16:24:46,512 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5651.82 MB 2025-02-14 16:24:46,512 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64340.62 MB 2025-02-14 16:24:46,512 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53647.25 MB 2025-02-14 16:24:46,512 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -10693.38 MB 2025-02-14 16:24:46,512 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 49253.55 MB 2025-02-14 16:24:46,643 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 16:24:46,643 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 16:24:46,643 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.13 seconds 2025-02-14 16:24:46,643 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:24:46,643 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40451.18 MB 2025-02-14 16:24:46,643 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34782.65 MB 2025-02-14 16:24:46,643 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -5668.53 MB 2025-02-14 16:24:46,643 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 53647.25 MB 2025-02-14 16:24:46,643 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 64317.55 MB 2025-02-14 16:24:46,643 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10670.31 MB 2025-02-14 16:24:46,643 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 56542.41 MB 2025-02-14 16:24:48,568 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 16:24:48,569 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 16:24:48,569 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.92 seconds 2025-02-14 16:24:48,569 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:24:48,569 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34782.65 MB 2025-02-14 16:24:48,569 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35313.49 MB 2025-02-14 16:24:48,569 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 16:24:48,569 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64317.55 MB 2025-02-14 16:24:48,569 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43218.11 MB 2025-02-14 16:24:48,569 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21099.45 MB 2025-02-14 16:24:48,569 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39292.82 MB 2025-02-14 16:24:48,582 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 16:24:48,583 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 16:24:48,583 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 16:24:48,583 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:24:48,583 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35313.49 MB 2025-02-14 16:24:48,583 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 37203.01 MB 2025-02-14 16:24:48,583 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.52 MB 2025-02-14 16:24:48,583 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43218.11 MB 2025-02-14 16:24:48,583 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43218.11 MB 2025-02-14 16:24:48,583 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:24:48,583 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38620.44 MB 2025-02-14 16:24:48,792 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 16:24:48,792 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 16:24:48,792 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 16:24:48,792 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:24:48,792 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37203.01 MB 2025-02-14 16:24:48,792 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39444.87 MB 2025-02-14 16:24:48,792 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 16:24:48,792 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43218.11 MB 2025-02-14 16:24:48,792 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47464.84 MB 2025-02-14 16:24:48,792 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4246.73 MB 2025-02-14 16:24:48,792 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44989.15 MB 2025-02-14 16:24:48,793 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 16:24:48,793 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 16:24:48,793 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 16:24:48,793 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:24:48,793 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35313.49 MB 2025-02-14 16:24:48,793 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39444.87 MB 2025-02-14 16:24:48,793 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.38 MB 2025-02-14 16:24:48,793 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43218.11 MB 2025-02-14 16:24:48,793 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47464.84 MB 2025-02-14 16:24:48,793 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4246.73 MB 2025-02-14 16:24:48,793 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44989.15 MB 2025-02-14 16:24:48,951 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 16:24:48,951 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 16:24:48,951 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 16:24:48,951 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:24:48,951 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40152.66 MB 2025-02-14 16:24:48,951 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40919.66 MB 2025-02-14 16:24:48,951 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 16:24:48,951 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47464.84 MB 2025-02-14 16:24:48,951 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47882.17 MB 2025-02-14 16:24:48,951 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 16:24:48,951 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41627.45 MB 2025-02-14 16:24:48,968 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 16:24:48,968 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 16:24:48,968 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 16:24:48,968 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:24:48,968 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41332.55 MB 2025-02-14 16:24:48,968 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41539.14 MB 2025-02-14 16:24:48,968 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.60 MB 2025-02-14 16:24:48,968 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47882.17 MB 2025-02-14 16:24:48,968 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47882.17 MB 2025-02-14 16:24:48,968 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:24:48,968 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41733.96 MB 2025-02-14 16:24:48,969 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 16:24:48,969 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 16:24:48,970 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.06 seconds 2025-02-14 16:24:48,970 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:24:48,970 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29235.27 MB 2025-02-14 16:24:48,970 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41739.75 MB 2025-02-14 16:24:48,970 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12504.47 MB 2025-02-14 16:24:48,970 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 64340.62 MB 2025-02-14 16:24:48,970 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47882.17 MB 2025-02-14 16:24:48,970 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -16458.45 MB 2025-02-14 16:24:48,970 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41739.75 MB 2025-02-14 16:24:49,234 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 16:24:49,234 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 16:24:49,234 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 16:24:49,234 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:24:49,234 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41739.75 MB 2025-02-14 16:24:49,234 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41839.98 MB 2025-02-14 16:24:49,234 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.23 MB 2025-02-14 16:24:49,234 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47882.17 MB 2025-02-14 16:24:49,234 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47882.17 MB 2025-02-14 16:24:49,234 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:24:49,234 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42441.38 MB 2025-02-14 16:24:49,252 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8143, cut from 8145 2025-02-14 16:24:49,252 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 16:24:49,258 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 16:24:49,258 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 16:24:49,258 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 16:24:49,258 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:24:49,258 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30497.73 MB 2025-02-14 16:24:49,258 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34682.47 MB 2025-02-14 16:24:49,258 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4184.74 MB 2025-02-14 16:24:49,258 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 47882.17 MB 2025-02-14 16:24:49,258 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 47882.17 MB 2025-02-14 16:24:49,258 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:24:49,258 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38866.69 MB 2025-02-14 16:24:49,410 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7935] 2025-02-14 16:24:49,411 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:24:49,411 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:24:49,412 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:24:49,412 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 16:24:49,416 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 16:24:49,417 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:24:49,417 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 16:24:49,417 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 16:24:49,418 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:24:49,418 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:24:49,419 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:24:49,419 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:24:49,424 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 16:24:49,425 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:24:49,425 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:24:49,425 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:24:49,425 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:24:49,425 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 16:24:49,426 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:24:49,426 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:24:49,426 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:24:49,426 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 16:24:49,426 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 16:24:49,427 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:24:49,427 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:24:49,432 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:24:49,432 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:24:49,433 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:24:49,433 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:24:49,434 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:24:49,434 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:24:49,441 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:24:49,441 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:26:12,870 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:26:12,870 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:26:12,875 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 16:26:12,876 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:26:12,876 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1727, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 16:26:12,877 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:26:12,877 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1727, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 16:26:39,478 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 16:26:39,478 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 16:26:39,478 - resource_logging.py:150 - __exit__ - DEBUG - Time: 26.59 seconds 2025-02-14 16:26:39,478 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:26:39,478 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35826.94 MB 2025-02-14 16:26:39,478 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41938.69 MB 2025-02-14 16:26:39,478 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6111.76 MB 2025-02-14 16:26:39,478 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56151.24 MB 2025-02-14 16:26:39,478 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49817.85 MB 2025-02-14 16:26:39,478 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6333.40 MB 2025-02-14 16:26:39,478 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 50816.86 MB 2025-02-14 16:26:39,589 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 16:26:39,589 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 16:26:39,590 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 16:26:39,590 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:26:39,590 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41938.69 MB 2025-02-14 16:26:39,590 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35580.20 MB 2025-02-14 16:26:39,590 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6358.49 MB 2025-02-14 16:26:39,590 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49817.85 MB 2025-02-14 16:26:39,590 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 65674.41 MB 2025-02-14 16:26:39,590 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 15856.57 MB 2025-02-14 16:26:39,590 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 59599.92 MB 2025-02-14 16:26:41,526 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 16:26:41,526 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 16:26:41,526 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.93 seconds 2025-02-14 16:26:41,526 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:26:41,526 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35580.20 MB 2025-02-14 16:26:41,526 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36111.05 MB 2025-02-14 16:26:41,526 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 16:26:41,526 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 65674.41 MB 2025-02-14 16:26:41,526 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43704.65 MB 2025-02-14 16:26:41,526 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -21969.76 MB 2025-02-14 16:26:41,526 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40090.38 MB 2025-02-14 16:26:41,539 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 16:26:41,539 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 16:26:41,539 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 16:26:41,540 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:26:41,540 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36111.05 MB 2025-02-14 16:26:41,540 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38000.32 MB 2025-02-14 16:26:41,540 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.27 MB 2025-02-14 16:26:41,540 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43704.65 MB 2025-02-14 16:26:41,540 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43704.65 MB 2025-02-14 16:26:41,540 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:26:41,540 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39417.75 MB 2025-02-14 16:26:41,760 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 16:26:41,760 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 16:26:41,760 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 16:26:41,760 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:26:41,760 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38000.32 MB 2025-02-14 16:26:41,760 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40242.17 MB 2025-02-14 16:26:41,760 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 16:26:41,760 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43704.65 MB 2025-02-14 16:26:41,760 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48895.10 MB 2025-02-14 16:26:41,760 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 16:26:41,760 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45786.46 MB 2025-02-14 16:26:41,761 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 16:26:41,761 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 16:26:41,761 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 16:26:41,761 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:26:41,761 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36111.05 MB 2025-02-14 16:26:41,761 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40242.17 MB 2025-02-14 16:26:41,761 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.13 MB 2025-02-14 16:26:41,761 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43704.65 MB 2025-02-14 16:26:41,761 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48895.10 MB 2025-02-14 16:26:41,761 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5190.45 MB 2025-02-14 16:26:41,761 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 45786.46 MB 2025-02-14 16:26:41,928 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 16:26:41,928 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 16:26:41,928 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 16:26:41,928 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:26:41,928 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 40949.96 MB 2025-02-14 16:26:41,928 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 41716.96 MB 2025-02-14 16:26:41,928 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 16:26:41,928 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48895.10 MB 2025-02-14 16:26:41,928 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49310.33 MB 2025-02-14 16:26:41,928 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 16:26:41,928 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42424.75 MB 2025-02-14 16:26:41,945 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 16:26:41,945 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 16:26:41,945 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 16:26:41,945 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:26:41,945 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42129.85 MB 2025-02-14 16:26:41,945 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42336.67 MB 2025-02-14 16:26:41,945 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 206.82 MB 2025-02-14 16:26:41,945 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49310.33 MB 2025-02-14 16:26:41,945 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49310.33 MB 2025-02-14 16:26:41,945 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:26:41,945 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42533.51 MB 2025-02-14 16:26:41,946 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 16:26:41,946 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 16:26:41,946 - resource_logging.py:150 - __exit__ - DEBUG - Time: 29.07 seconds 2025-02-14 16:26:41,946 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:26:41,946 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29809.93 MB 2025-02-14 16:26:41,946 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42537.45 MB 2025-02-14 16:26:41,946 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12727.52 MB 2025-02-14 16:26:41,946 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 56151.24 MB 2025-02-14 16:26:41,946 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49310.33 MB 2025-02-14 16:26:41,946 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6840.91 MB 2025-02-14 16:26:41,946 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42537.45 MB 2025-02-14 16:26:42,212 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 16:26:42,212 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 16:26:42,212 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 16:26:42,212 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:26:42,212 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42537.45 MB 2025-02-14 16:26:42,212 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42637.77 MB 2025-02-14 16:26:42,212 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.32 MB 2025-02-14 16:26:42,212 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49310.33 MB 2025-02-14 16:26:42,212 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49310.33 MB 2025-02-14 16:26:42,212 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:26:42,212 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43239.68 MB 2025-02-14 16:26:42,230 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8150, cut from 8152 2025-02-14 16:26:42,230 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 16:26:42,236 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 16:26:42,236 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 16:26:42,236 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 16:26:42,236 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:26:42,236 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31072.56 MB 2025-02-14 16:26:42,236 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35260.89 MB 2025-02-14 16:26:42,236 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4188.33 MB 2025-02-14 16:26:42,236 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49310.33 MB 2025-02-14 16:26:42,236 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53498.35 MB 2025-02-14 16:26:42,236 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4188.01 MB 2025-02-14 16:26:42,236 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39448.90 MB 2025-02-14 16:26:42,405 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7942] 2025-02-14 16:26:42,406 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:26:42,406 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:26:42,407 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:26:42,407 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 16:26:42,412 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 16:26:42,413 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:26:42,413 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 16:26:42,413 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 16:26:42,414 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:26:42,414 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:26:42,415 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:26:42,415 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:26:42,421 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 16:26:42,421 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:26:42,421 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:26:42,422 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:26:42,422 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:26:42,422 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 16:26:42,422 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:26:42,422 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:26:42,423 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:26:42,423 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 16:26:42,423 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 16:26:42,424 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:26:42,424 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:26:42,429 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:26:42,429 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:26:42,430 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:26:42,430 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:26:42,432 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:26:42,432 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:26:42,440 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:26:42,440 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:27:22,734 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:27:22,734 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:27:22,739 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 16:27:22,741 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:27:22,741 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 1781, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 16:27:22,741 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:27:22,741 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 1781, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 16:27:50,291 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 16:27:50,292 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 16:27:50,292 - resource_logging.py:150 - __exit__ - DEBUG - Time: 27.54 seconds 2025-02-14 16:27:50,292 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:27:50,292 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36324.94 MB 2025-02-14 16:27:50,292 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42627.80 MB 2025-02-14 16:27:50,292 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6302.86 MB 2025-02-14 16:27:50,292 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61889.05 MB 2025-02-14 16:27:50,292 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54318.33 MB 2025-02-14 16:27:50,292 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -7570.72 MB 2025-02-14 16:27:50,292 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 51458.63 MB 2025-02-14 16:27:50,395 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 16:27:50,395 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 16:27:50,395 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 16:27:50,395 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:27:50,395 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42627.80 MB 2025-02-14 16:27:50,395 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35982.66 MB 2025-02-14 16:27:50,395 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -6645.14 MB 2025-02-14 16:27:50,395 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 54318.33 MB 2025-02-14 16:27:50,395 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 67383.59 MB 2025-02-14 16:27:50,395 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 13065.26 MB 2025-02-14 16:27:50,395 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 61079.18 MB 2025-02-14 16:27:52,332 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 16:27:52,333 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 16:27:52,333 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.94 seconds 2025-02-14 16:27:52,333 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:27:52,333 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 35982.66 MB 2025-02-14 16:27:52,333 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36513.50 MB 2025-02-14 16:27:52,333 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 16:27:52,333 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 67383.59 MB 2025-02-14 16:27:52,333 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43236.98 MB 2025-02-14 16:27:52,333 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -24146.61 MB 2025-02-14 16:27:52,333 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 40492.83 MB 2025-02-14 16:27:52,346 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 16:27:52,346 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 16:27:52,346 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 16:27:52,346 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:27:52,346 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36513.50 MB 2025-02-14 16:27:52,346 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38402.71 MB 2025-02-14 16:27:52,346 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.21 MB 2025-02-14 16:27:52,346 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43236.98 MB 2025-02-14 16:27:52,346 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43236.98 MB 2025-02-14 16:27:52,346 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:27:52,346 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39820.14 MB 2025-02-14 16:27:52,562 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 16:27:52,562 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 16:27:52,562 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 16:27:52,562 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:27:52,562 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38402.71 MB 2025-02-14 16:27:52,562 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40644.56 MB 2025-02-14 16:27:52,562 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 16:27:52,562 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43236.98 MB 2025-02-14 16:27:52,562 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48899.29 MB 2025-02-14 16:27:52,562 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 16:27:52,562 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46188.84 MB 2025-02-14 16:27:52,562 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 16:27:52,562 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 16:27:52,562 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.23 seconds 2025-02-14 16:27:52,562 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:27:52,562 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36513.50 MB 2025-02-14 16:27:52,562 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 40644.56 MB 2025-02-14 16:27:52,563 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.06 MB 2025-02-14 16:27:52,563 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 43236.98 MB 2025-02-14 16:27:52,563 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 48899.29 MB 2025-02-14 16:27:52,563 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 5662.31 MB 2025-02-14 16:27:52,563 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 46188.84 MB 2025-02-14 16:27:52,729 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 16:27:52,729 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 16:27:52,729 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.16 seconds 2025-02-14 16:27:52,729 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:27:52,729 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 41352.35 MB 2025-02-14 16:27:52,729 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42119.35 MB 2025-02-14 16:27:52,729 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 16:27:52,729 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 48899.29 MB 2025-02-14 16:27:52,729 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49316.63 MB 2025-02-14 16:27:52,729 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 417.33 MB 2025-02-14 16:27:52,729 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42827.14 MB 2025-02-14 16:27:52,747 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 16:27:52,747 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 16:27:52,747 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 16:27:52,747 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:27:52,747 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42532.24 MB 2025-02-14 16:27:52,747 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42737.84 MB 2025-02-14 16:27:52,747 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.59 MB 2025-02-14 16:27:52,747 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49316.63 MB 2025-02-14 16:27:52,747 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49316.63 MB 2025-02-14 16:27:52,747 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:27:52,747 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42951.69 MB 2025-02-14 16:27:52,748 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 16:27:52,748 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 16:27:52,748 - resource_logging.py:150 - __exit__ - DEBUG - Time: 30.00 seconds 2025-02-14 16:27:52,748 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:27:52,748 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30119.80 MB 2025-02-14 16:27:52,748 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 42938.61 MB 2025-02-14 16:27:52,748 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12818.82 MB 2025-02-14 16:27:52,748 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 61889.05 MB 2025-02-14 16:27:52,748 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49316.63 MB 2025-02-14 16:27:52,748 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12572.43 MB 2025-02-14 16:27:52,748 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42951.69 MB 2025-02-14 16:27:53,015 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 16:27:53,015 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 16:27:53,015 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 16:27:53,016 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:27:53,016 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 42938.61 MB 2025-02-14 16:27:53,016 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 43038.71 MB 2025-02-14 16:27:53,016 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.10 MB 2025-02-14 16:27:53,016 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49316.63 MB 2025-02-14 16:27:53,016 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 49316.63 MB 2025-02-14 16:27:53,016 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:27:53,016 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43639.30 MB 2025-02-14 16:27:53,033 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8132, cut from 8134 2025-02-14 16:27:53,033 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 16:27:53,039 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 16:27:53,040 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 16:27:53,040 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 16:27:53,040 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:27:53,040 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31381.98 MB 2025-02-14 16:27:53,040 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 35561.08 MB 2025-02-14 16:27:53,040 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4179.09 MB 2025-02-14 16:27:53,040 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 49316.63 MB 2025-02-14 16:27:53,040 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 53496.25 MB 2025-02-14 16:27:53,040 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4179.62 MB 2025-02-14 16:27:53,040 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39740.70 MB 2025-02-14 16:27:53,203 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7924] 2025-02-14 16:27:53,204 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:27:53,204 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:27:53,205 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:27:53,205 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 16:27:53,210 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 16:27:53,211 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:27:53,211 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 16:27:53,211 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 16:27:53,211 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:27:53,211 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:27:53,212 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:27:53,212 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:27:53,218 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 16:27:53,218 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:27:53,218 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:27:53,219 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:27:53,219 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:27:53,219 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 16:27:53,219 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:27:53,219 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:27:53,220 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:27:53,220 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 16:27:53,220 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 16:27:53,220 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:27:53,220 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:27:53,227 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:27:53,227 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:27:53,229 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:27:53,229 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:27:53,231 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:27:53,231 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:27:53,239 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:27:53,239 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:28:04,257 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:28:04,258 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:28:04,262 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 16:28:04,263 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:28:04,263 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 823, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 16:28:04,264 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:28:04,264 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 823, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 16:28:17,160 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 16:28:17,160 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 16:28:17,160 - resource_logging.py:150 - __exit__ - DEBUG - Time: 12.89 seconds 2025-02-14 16:28:17,160 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:28:17,160 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29771.17 MB 2025-02-14 16:28:17,160 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32683.72 MB 2025-02-14 16:28:17,160 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2912.55 MB 2025-02-14 16:28:17,160 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62008.59 MB 2025-02-14 16:28:17,160 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39164.31 MB 2025-02-14 16:28:17,160 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22844.28 MB 2025-02-14 16:28:17,160 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 41507.47 MB 2025-02-14 16:28:17,193 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 16:28:17,193 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 16:28:17,193 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.03 seconds 2025-02-14 16:28:17,193 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:28:17,193 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32683.72 MB 2025-02-14 16:28:17,193 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29775.62 MB 2025-02-14 16:28:17,193 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2908.10 MB 2025-02-14 16:28:17,193 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39164.31 MB 2025-02-14 16:28:17,193 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39164.31 MB 2025-02-14 16:28:17,193 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:28:17,193 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35630.07 MB 2025-02-14 16:28:18,201 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 16:28:18,201 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 16:28:18,201 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.01 seconds 2025-02-14 16:28:18,201 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:28:18,201 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29775.62 MB 2025-02-14 16:28:18,201 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 30051.66 MB 2025-02-14 16:28:18,201 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 276.04 MB 2025-02-14 16:28:18,201 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39164.31 MB 2025-02-14 16:28:18,201 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39164.31 MB 2025-02-14 16:28:18,201 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:28:18,201 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34030.99 MB 2025-02-14 16:28:18,210 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 16:28:18,210 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 16:28:18,210 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 16:28:18,210 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:28:18,210 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30051.66 MB 2025-02-14 16:28:18,210 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31033.98 MB 2025-02-14 16:28:18,210 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 982.32 MB 2025-02-14 16:28:18,210 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39164.31 MB 2025-02-14 16:28:18,210 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39164.31 MB 2025-02-14 16:28:18,210 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:28:18,210 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31771.04 MB 2025-02-14 16:28:18,321 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 16:28:18,321 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 16:28:18,321 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 16:28:18,321 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:28:18,321 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31033.98 MB 2025-02-14 16:28:18,321 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32200.30 MB 2025-02-14 16:28:18,321 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1166.32 MB 2025-02-14 16:28:18,321 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39164.31 MB 2025-02-14 16:28:18,321 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39164.31 MB 2025-02-14 16:28:18,321 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:28:18,321 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35083.29 MB 2025-02-14 16:28:18,321 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 16:28:18,321 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 16:28:18,321 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.12 seconds 2025-02-14 16:28:18,321 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:28:18,321 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 30051.66 MB 2025-02-14 16:28:18,321 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32200.30 MB 2025-02-14 16:28:18,321 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2148.64 MB 2025-02-14 16:28:18,322 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39164.31 MB 2025-02-14 16:28:18,322 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39164.31 MB 2025-02-14 16:28:18,322 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:28:18,322 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35083.29 MB 2025-02-14 16:28:18,408 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 16:28:18,408 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 16:28:18,408 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 16:28:18,408 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:28:18,408 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32568.35 MB 2025-02-14 16:28:18,408 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32967.19 MB 2025-02-14 16:28:18,408 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 398.84 MB 2025-02-14 16:28:18,408 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39164.31 MB 2025-02-14 16:28:18,408 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39378.22 MB 2025-02-14 16:28:18,408 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 213.91 MB 2025-02-14 16:28:18,408 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33337.51 MB 2025-02-14 16:28:18,420 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 16:28:18,420 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 16:28:18,420 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 16:28:18,420 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:28:18,420 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 33181.90 MB 2025-02-14 16:28:18,420 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33383.27 MB 2025-02-14 16:28:18,420 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 201.37 MB 2025-02-14 16:28:18,420 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39378.22 MB 2025-02-14 16:28:18,420 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39380.32 MB 2025-02-14 16:28:18,420 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 2.10 MB 2025-02-14 16:28:18,420 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33409.56 MB 2025-02-14 16:28:18,421 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 16:28:18,421 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 16:28:18,421 - resource_logging.py:150 - __exit__ - DEBUG - Time: 14.16 seconds 2025-02-14 16:28:18,421 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:28:18,421 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26903.77 MB 2025-02-14 16:28:18,421 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33583.92 MB 2025-02-14 16:28:18,421 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 6680.15 MB 2025-02-14 16:28:18,421 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 62008.59 MB 2025-02-14 16:28:18,421 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39380.32 MB 2025-02-14 16:28:18,422 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -22628.27 MB 2025-02-14 16:28:18,422 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33583.92 MB 2025-02-14 16:28:18,688 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 16:28:18,688 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 16:28:18,688 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 16:28:18,688 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:28:18,688 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27556.30 MB 2025-02-14 16:28:18,688 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27656.56 MB 2025-02-14 16:28:18,688 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.26 MB 2025-02-14 16:28:18,688 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39380.32 MB 2025-02-14 16:28:18,688 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39380.32 MB 2025-02-14 16:28:18,688 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:28:18,688 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28258.11 MB 2025-02-14 16:28:18,706 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8145, cut from 8147 2025-02-14 16:28:18,706 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 16:28:18,712 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 16:28:18,712 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 16:28:18,712 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 16:28:18,712 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:28:18,712 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27656.56 MB 2025-02-14 16:28:18,712 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 31842.32 MB 2025-02-14 16:28:18,712 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4185.76 MB 2025-02-14 16:28:18,712 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39380.32 MB 2025-02-14 16:28:18,712 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43566.24 MB 2025-02-14 16:28:18,712 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4185.92 MB 2025-02-14 16:28:18,712 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36028.24 MB 2025-02-14 16:28:18,876 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7937] 2025-02-14 16:28:18,877 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:28:18,877 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:28:18,878 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:28:18,878 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 16:28:18,883 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 16:28:18,884 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:28:18,884 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 16:28:18,884 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 16:28:18,885 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:28:18,885 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:28:18,885 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:28:18,885 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:28:18,892 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 16:28:18,893 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:28:18,893 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:28:18,893 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:28:18,893 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:28:18,893 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 16:28:18,894 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:28:18,894 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:28:18,895 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:28:18,895 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 16:28:18,895 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 16:28:18,895 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:28:18,895 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:28:18,902 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:28:18,902 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:28:18,904 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:28:18,904 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:28:18,906 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:28:18,906 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:28:18,915 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:28:18,915 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:29:05,991 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:29:05,991 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:29:05,996 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 16:29:05,997 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:29:05,997 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 186, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 16:29:05,998 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:29:05,998 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 186, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 16:29:08,849 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 16:29:08,849 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 16:29:08,849 - resource_logging.py:150 - __exit__ - DEBUG - Time: 2.85 seconds 2025-02-14 16:29:08,849 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:29:08,849 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25454.18 MB 2025-02-14 16:29:08,849 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26112.42 MB 2025-02-14 16:29:08,849 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 658.24 MB 2025-02-14 16:29:08,849 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52200.21 MB 2025-02-14 16:29:08,849 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39285.95 MB 2025-02-14 16:29:08,849 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12914.26 MB 2025-02-14 16:29:08,849 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 34925.55 MB 2025-02-14 16:29:08,863 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 16:29:08,863 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 16:29:08,863 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 16:29:08,863 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:29:08,863 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26112.42 MB 2025-02-14 16:29:08,863 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26431.34 MB 2025-02-14 16:29:08,863 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 318.92 MB 2025-02-14 16:29:08,863 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39285.95 MB 2025-02-14 16:29:08,863 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39285.95 MB 2025-02-14 16:29:08,863 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:29:08,863 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28760.44 MB 2025-02-14 16:29:09,748 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 16:29:09,748 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 16:29:09,748 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.88 seconds 2025-02-14 16:29:09,748 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:29:09,749 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26431.34 MB 2025-02-14 16:29:09,749 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 26678.18 MB 2025-02-14 16:29:09,749 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 246.84 MB 2025-02-14 16:29:09,749 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39285.95 MB 2025-02-14 16:29:09,749 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39285.95 MB 2025-02-14 16:29:09,749 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:29:09,749 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30601.78 MB 2025-02-14 16:29:09,757 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 16:29:09,757 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 16:29:09,757 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 16:29:09,757 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:29:09,757 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26678.12 MB 2025-02-14 16:29:09,757 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 27556.54 MB 2025-02-14 16:29:09,757 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 878.42 MB 2025-02-14 16:29:09,757 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39285.95 MB 2025-02-14 16:29:09,757 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39285.95 MB 2025-02-14 16:29:09,757 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:29:09,757 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 28215.65 MB 2025-02-14 16:29:09,863 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 16:29:09,863 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 16:29:09,863 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.10 seconds 2025-02-14 16:29:09,863 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:29:09,863 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27556.54 MB 2025-02-14 16:29:09,863 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28599.03 MB 2025-02-14 16:29:09,863 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1042.50 MB 2025-02-14 16:29:09,863 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39285.95 MB 2025-02-14 16:29:09,864 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39285.95 MB 2025-02-14 16:29:09,864 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:29:09,864 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31177.09 MB 2025-02-14 16:29:09,864 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 16:29:09,864 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 16:29:09,864 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.11 seconds 2025-02-14 16:29:09,864 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:29:09,864 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 26678.12 MB 2025-02-14 16:29:09,864 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 28599.03 MB 2025-02-14 16:29:09,864 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1920.92 MB 2025-02-14 16:29:09,864 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39285.95 MB 2025-02-14 16:29:09,864 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39285.95 MB 2025-02-14 16:29:09,864 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:29:09,864 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 31177.09 MB 2025-02-14 16:29:09,944 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 16:29:09,944 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 16:29:09,944 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.08 seconds 2025-02-14 16:29:09,944 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:29:09,944 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28928.16 MB 2025-02-14 16:29:09,944 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29284.81 MB 2025-02-14 16:29:09,944 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 356.66 MB 2025-02-14 16:29:09,944 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39285.95 MB 2025-02-14 16:29:09,944 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39476.79 MB 2025-02-14 16:29:09,944 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 190.84 MB 2025-02-14 16:29:09,944 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29618.30 MB 2025-02-14 16:29:09,954 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 16:29:09,954 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 16:29:09,954 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 16:29:09,954 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:29:09,954 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29476.81 MB 2025-02-14 16:29:09,954 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29678.19 MB 2025-02-14 16:29:09,954 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 201.37 MB 2025-02-14 16:29:09,954 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39476.79 MB 2025-02-14 16:29:09,954 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39480.98 MB 2025-02-14 16:29:09,954 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4.19 MB 2025-02-14 16:29:09,954 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29715.29 MB 2025-02-14 16:29:09,955 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 16:29:09,955 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 16:29:09,955 - resource_logging.py:150 - __exit__ - DEBUG - Time: 3.95 seconds 2025-02-14 16:29:09,955 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:29:09,955 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 24806.14 MB 2025-02-14 16:29:09,955 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29879.26 MB 2025-02-14 16:29:09,955 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 5073.12 MB 2025-02-14 16:29:09,955 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52200.21 MB 2025-02-14 16:29:09,955 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39480.98 MB 2025-02-14 16:29:09,955 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -12719.23 MB 2025-02-14 16:29:09,955 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 29879.26 MB 2025-02-14 16:29:10,216 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 16:29:10,216 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 16:29:10,216 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 16:29:10,216 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:29:10,216 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 29879.26 MB 2025-02-14 16:29:10,216 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29979.73 MB 2025-02-14 16:29:10,216 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.47 MB 2025-02-14 16:29:10,216 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39480.98 MB 2025-02-14 16:29:10,216 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39480.98 MB 2025-02-14 16:29:10,216 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:29:10,216 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 30582.53 MB 2025-02-14 16:29:10,234 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8162, cut from 8164 2025-02-14 16:29:10,234 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The video rate for this video is 2 ('] 2025-02-14 16:29:10,240 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 16:29:10,240 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 16:29:10,240 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 16:29:10,240 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:29:10,240 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 25500.94 MB 2025-02-14 16:29:10,240 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 29695.42 MB 2025-02-14 16:29:10,240 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4194.49 MB 2025-02-14 16:29:10,240 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39480.98 MB 2025-02-14 16:29:10,240 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 43675.29 MB 2025-02-14 16:29:10,240 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4194.30 MB 2025-02-14 16:29:10,240 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 33889.73 MB 2025-02-14 16:29:10,402 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7954] 2025-02-14 16:29:10,403 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:29:10,403 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:29:10,404 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:29:10,404 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 16:29:10,409 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 16:29:10,410 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:29:10,410 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 16:29:10,410 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The video rate for this video is 2 ('] 2025-02-14 16:29:10,410 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:29:10,410 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:29:10,411 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:29:10,411 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:29:10,417 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 16:29:10,417 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:29:10,417 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:29:10,418 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:29:10,418 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:29:10,418 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 16:29:10,418 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:29:10,418 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:29:10,419 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:29:10,419 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 16:29:10,419 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 16:29:10,419 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:29:10,419 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:29:10,424 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:29:10,424 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:29:10,426 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:29:10,426 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:29:10,427 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:29:10,427 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:29:10,435 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:29:10,436 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:31:34,285 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:31:34,285 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:31:34,290 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 16:31:34,291 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:31:34,291 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 972, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 16:31:34,292 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:31:34,292 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 972, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 16:31:49,154 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 16:31:49,154 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 16:31:49,154 - resource_logging.py:150 - __exit__ - DEBUG - Time: 14.86 seconds 2025-02-14 16:31:49,154 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:31:49,154 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 31052.88 MB 2025-02-14 16:31:49,154 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34492.73 MB 2025-02-14 16:31:49,154 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 3439.85 MB 2025-02-14 16:31:49,154 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52430.90 MB 2025-02-14 16:31:49,155 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 39407.58 MB 2025-02-14 16:31:49,155 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -13023.31 MB 2025-02-14 16:31:49,155 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 43468.65 MB 2025-02-14 16:31:49,215 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 16:31:49,215 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 16:31:49,215 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.06 seconds 2025-02-14 16:31:49,215 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:31:49,215 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34492.73 MB 2025-02-14 16:31:49,215 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32142.10 MB 2025-02-14 16:31:49,215 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: -2350.64 MB 2025-02-14 16:31:49,215 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 39407.58 MB 2025-02-14 16:31:49,215 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 50025.46 MB 2025-02-14 16:31:49,215 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 10617.88 MB 2025-02-14 16:31:49,215 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 44983.30 MB 2025-02-14 16:31:51,121 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:siglip 2025-02-14 16:31:51,121 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 877 2025-02-14 16:31:51,121 - resource_logging.py:150 - __exit__ - DEBUG - Time: 1.90 seconds 2025-02-14 16:31:51,122 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:31:51,122 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32142.10 MB 2025-02-14 16:31:51,122 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 32672.94 MB 2025-02-14 16:31:51,122 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 530.84 MB 2025-02-14 16:31:51,122 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 50025.46 MB 2025-02-14 16:31:51,122 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40823.16 MB 2025-02-14 16:31:51,122 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -9202.30 MB 2025-02-14 16:31:51,122 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 36652.27 MB 2025-02-14 16:31:51,135 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> mm_projector_aux_0/1 2025-02-14 16:31:51,135 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 918 2025-02-14 16:31:51,135 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 16:31:51,135 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:31:51,135 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32672.94 MB 2025-02-14 16:31:51,135 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 34562.23 MB 2025-02-14 16:31:51,135 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 1889.29 MB 2025-02-14 16:31:51,135 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40823.16 MB 2025-02-14 16:31:51,135 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 40823.16 MB 2025-02-14 16:31:51,135 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:31:51,135 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 35979.66 MB 2025-02-14 16:31:51,342 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA -> query_group 2025-02-14 16:31:51,342 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 936 2025-02-14 16:31:51,342 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.21 seconds 2025-02-14 16:31:51,342 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:31:51,342 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 34562.23 MB 2025-02-14 16:31:51,342 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36804.09 MB 2025-02-14 16:31:51,342 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 2241.86 MB 2025-02-14 16:31:51,342 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40823.16 MB 2025-02-14 16:31:51,342 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45541.75 MB 2025-02-14 16:31:51,342 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 16:31:51,342 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42348.37 MB 2025-02-14 16:31:51,343 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> SVA 2025-02-14 16:31:51,343 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 913 2025-02-14 16:31:51,343 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.22 seconds 2025-02-14 16:31:51,343 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:31:51,343 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 32672.94 MB 2025-02-14 16:31:51,343 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 36804.09 MB 2025-02-14 16:31:51,343 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4131.15 MB 2025-02-14 16:31:51,343 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 40823.16 MB 2025-02-14 16:31:51,343 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45541.75 MB 2025-02-14 16:31:51,343 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 4718.59 MB 2025-02-14 16:31:51,343 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 42348.37 MB 2025-02-14 16:31:51,499 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> rearrange_vision_tower+padding 2025-02-14 16:31:51,499 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1079 2025-02-14 16:31:51,499 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.15 seconds 2025-02-14 16:31:51,499 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:31:51,499 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 37511.88 MB 2025-02-14 16:31:51,499 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38278.88 MB 2025-02-14 16:31:51,499 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 767.00 MB 2025-02-14 16:31:51,499 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45541.75 MB 2025-02-14 16:31:51,499 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45956.99 MB 2025-02-14 16:31:51,499 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 415.24 MB 2025-02-14 16:31:51,499 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 38986.67 MB 2025-02-14 16:31:51,516 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> Embedding+Cross-modal+STC 2025-02-14 16:31:51,516 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 1380 2025-02-14 16:31:51,516 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.01 seconds 2025-02-14 16:31:51,516 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:31:51,516 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 38691.77 MB 2025-02-14 16:31:51,516 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 38897.70 MB 2025-02-14 16:31:51,516 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 205.93 MB 2025-02-14 16:31:51,516 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45956.99 MB 2025-02-14 16:31:51,516 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45956.99 MB 2025-02-14 16:31:51,516 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:31:51,516 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39095.65 MB 2025-02-14 16:31:51,518 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 16:31:51,518 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 16:31:51,518 - resource_logging.py:150 - __exit__ - DEBUG - Time: 17.22 seconds 2025-02-14 16:31:51,518 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:31:51,518 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 27666.35 MB 2025-02-14 16:31:51,518 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39098.21 MB 2025-02-14 16:31:51,518 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 11431.85 MB 2025-02-14 16:31:51,518 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 52430.90 MB 2025-02-14 16:31:51,518 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45956.99 MB 2025-02-14 16:31:51,518 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6473.91 MB 2025-02-14 16:31:51,518 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39098.21 MB 2025-02-14 16:31:51,779 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> model.forward 2025-02-14 16:31:51,779 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 390 2025-02-14 16:31:51,779 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.26 seconds 2025-02-14 16:31:51,779 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:31:51,779 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 39098.21 MB 2025-02-14 16:31:51,779 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 39198.39 MB 2025-02-14 16:31:51,779 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 100.18 MB 2025-02-14 16:31:51,779 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45956.99 MB 2025-02-14 16:31:51,779 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 45956.99 MB 2025-02-14 16:31:51,779 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 0.00 MB 2025-02-14 16:31:51,779 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 39799.50 MB 2025-02-14 16:31:51,797 - cambrian_llama.py:481 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): Found assistant token at index 8139, cut from 8141 2025-02-14 16:31:51,797 - cambrian_llama.py:487 - forward - INFO - In CambrianLlamaForCausalLM.forward(): Decoded assistant outputs: ['The final rate for this video is 2 ('] 2025-02-14 16:31:51,803 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> lm_head, logits 2025-02-14 16:31:51,803 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 456 2025-02-14 16:31:51,803 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.02 seconds 2025-02-14 16:31:51,803 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:31:51,803 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 28928.71 MB 2025-02-14 16:31:51,803 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 33111.40 MB 2025-02-14 16:31:51,803 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 4182.69 MB 2025-02-14 16:31:51,803 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 45956.99 MB 2025-02-14 16:31:51,803 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 54324.63 MB 2025-02-14 16:31:51,803 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 8367.64 MB 2025-02-14 16:31:51,803 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 37293.57 MB 2025-02-14 16:31:51,958 - cambrian_llama.py:512 - forward - DEBUG - sample 0: correct range [16, 7931] 2025-02-14 16:31:51,959 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:31:51,959 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:31:51,960 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:31:51,960 - resource_logging.py:45 - debug_tensor - DEBUG - In CambrianLlamaForCausalLM.forward(): orig_labels: [torch.Size([1, 238]), torch.int64, cuda:0] 2025-02-14 16:31:51,964 - cambrian_llama.py:529 - forward - DEBUG - In CambrianLlamaForCausalLM.forward(): sample 0: output range: [225, 237] 2025-02-14 16:31:51,966 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:31:51,966 - resource_logging.py:45 - debug_tensor - DEBUG - outs: [torch.Size([1, 12]), torch.int64, cuda:0] 2025-02-14 16:31:51,966 - cambrian_llama.py:533 - forward - INFO - sample 0: decoded outputs: ['The final rate for this video is 2 ('] 2025-02-14 16:31:51,967 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:31:51,967 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:31:51,967 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:31:51,967 - resource_logging.py:45 - debug_tensor - DEBUG - In prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:31:51,973 - mm_trainer.py:995 - prediction_step - DEBUG - Assistant token at position 295 2025-02-14 16:31:51,974 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:31:51,974 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:31:51,974 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:31:51,974 - resource_logging.py:45 - debug_tensor - DEBUG - After prediction_step: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:31:51,974 - mm_trainer.py:767 - evaluation_loop - DEBUG - main_input_name: input_ids 2025-02-14 16:31:51,975 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:31:51,975 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['input_ids']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:31:51,975 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:31:51,975 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs['attention_mask']: [torch.Size([1, 8192]), torch.bool, cuda:0] 2025-02-14 16:31:51,975 - mm_trainer.py:773 - evaluation_loop - DEBUG - type(inputs_decode): 2025-02-14 16:31:51,976 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:31:51,976 - resource_logging.py:45 - debug_tensor - DEBUG - In evaluation_loop(): inputs_decode: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:31:51,982 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:31:51,982 - resource_logging.py:45 - debug_tensor - DEBUG - Before accelerator.pad_across_processes: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:31:51,982 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:31:51,982 - resource_logging.py:45 - debug_tensor - DEBUG - Before gather_function: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:31:51,983 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:31:51,983 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: logits: [torch.Size([1, 237, 128256]), torch.float32, cuda:0] 2025-02-14 16:31:51,992 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:31:51,992 - resource_logging.py:45 - debug_tensor - DEBUG - Add to all_preds: labels: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:33:05,358 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:33:05,359 - resource_logging.py:45 - debug_tensor - DEBUG - In compute_loss(): inputs['labels']: [torch.Size([1, 8192]), torch.int64, cuda:0] 2025-02-14 16:33:05,364 - mm_trainer.py:618 - compute_loss - DEBUG - In compute_loss(): assistant token at position 224 2025-02-14 16:33:05,365 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:33:05,366 - resource_logging.py:45 - debug_tensor - DEBUG - images_0: [torch.Size([1, 3569, 3, 384, 384]), torch.float32, cuda:0] 2025-02-14 16:33:05,366 - resource_logging.py:42 - debug_tensor - DEBUG - File: Unknown, Line: Unknown 2025-02-14 16:33:05,366 - resource_logging.py:45 - debug_tensor - DEBUG - images_1: [torch.Size([1, 3569, 3, 378, 378]), torch.float32, cuda:0] 2025-02-14 16:34:00,466 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> encode_images:dino 2025-02-14 16:34:00,466 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 856 2025-02-14 16:34:00,466 - resource_logging.py:150 - __exit__ - DEBUG - Time: 55.08 seconds 2025-02-14 16:34:00,466 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:34:00,466 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 49271.02 MB 2025-02-14 16:34:00,466 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 61902.17 MB 2025-02-14 16:34:00,466 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 12631.15 MB 2025-02-14 16:34:00,466 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 75440.85 MB 2025-02-14 16:34:00,466 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 68975.33 MB 2025-02-14 16:34:00,466 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: -6465.52 MB 2025-02-14 16:34:00,466 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 74532.66 MB 2025-02-14 16:34:00,769 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianMetaForCausalLM -> prepare_inputs_labels_for_multimodal -> select_frame 2025-02-14 16:34:00,769 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/cambrian_arch.py, Line: 862 2025-02-14 16:34:00,769 - resource_logging.py:150 - __exit__ - DEBUG - Time: 0.30 seconds 2025-02-14 16:34:00,769 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:34:00,769 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 61902.17 MB 2025-02-14 16:34:00,769 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 86816.98 MB 2025-02-14 16:34:00,769 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 24914.80 MB 2025-02-14 16:34:00,769 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 68975.33 MB 2025-02-14 16:34:00,769 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 90481.62 MB 2025-02-14 16:34:00,769 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 21506.29 MB 2025-02-14 16:34:00,769 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 93262.00 MB 2025-02-14 16:34:00,770 - resource_logging.py:148 - __exit__ - DEBUG - Section name: CambrianLlamaForCausalLM -> forward -> prepare_inputs_labels_for_multimodal 2025-02-14 16:34:00,770 - resource_logging.py:149 - __exit__ - DEBUG - File: /root/hcmus/LongVidLLaMA/longvu/language_model/cambrian_llama.py, Line: 309 2025-02-14 16:34:00,770 - resource_logging.py:150 - __exit__ - DEBUG - Time: 55.40 seconds 2025-02-14 16:34:00,770 - resource_logging.py:151 - __exit__ - DEBUG - Device: cuda:0 2025-02-14 16:34:00,770 - resource_logging.py:152 - __exit__ - DEBUG - Allocated before block: 36836.29 MB 2025-02-14 16:34:00,770 - resource_logging.py:153 - __exit__ - DEBUG - Allocated after block: 86816.98 MB 2025-02-14 16:34:00,770 - resource_logging.py:154 - __exit__ - DEBUG - Net allocated change: 49980.69 MB 2025-02-14 16:34:00,770 - resource_logging.py:155 - __exit__ - DEBUG - Reserved before block: 69321.36 MB 2025-02-14 16:34:00,770 - resource_logging.py:156 - __exit__ - DEBUG - Reserved after block: 90481.62 MB 2025-02-14 16:34:00,770 - resource_logging.py:157 - __exit__ - DEBUG - Net reserved change: 21160.26 MB 2025-02-14 16:34:00,770 - resource_logging.py:158 - __exit__ - DEBUG - Peak allocated: 93262.00 MB